Nov 01 22:19:47.654680 ip-10-0-128-55 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 22:19:47.654691 ip-10-0-128-55 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 22:19:47.654701 ip-10-0-128-55 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 22:19:47.655102 ip-10-0-128-55 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 22:19:57.903794 ip-10-0-128-55 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 22:19:57.903813 ip-10-0-128-55 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ab64b12d0c75484b9df7ef4571dc24a3 -- Nov 01 22:21:53.968406 ip-10-0-128-55 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:21:54.421271 ip-10-0-128-55 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:21:54.421271 ip-10-0-128-55 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:21:54.421271 ip-10-0-128-55 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:21:54.421271 ip-10-0-128-55 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:21:54.425837 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425757 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:21:54.425837 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425838 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425842 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425846 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425849 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425852 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425855 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425860 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425865 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425868 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425871 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425875 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425877 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425881 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425883 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425886 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425889 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425892 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425894 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425897 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:21:54.425907 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425900 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425902 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425905 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425908 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425910 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425913 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425916 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425919 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425922 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425929 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425932 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425935 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425937 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425940 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425943 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425945 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425948 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425950 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425953 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425956 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:21:54.426359 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425958 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425961 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425963 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425966 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425969 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425971 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425974 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425976 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425979 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425982 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425985 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425987 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425990 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425993 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425996 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.425999 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426002 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426004 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426007 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426009 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:21:54.426829 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426012 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426015 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426017 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426020 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426022 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426025 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426028 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426030 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426033 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426035 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426038 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426040 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426043 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426046 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426049 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426051 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426054 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426057 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426059 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426062 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:21:54.427321 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426066 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426069 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426071 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426076 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426080 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426083 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426086 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426089 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426091 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426094 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426096 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426099 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426102 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426104 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426107 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426122 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426125 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426128 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426131 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:21:54.427794 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426133 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426136 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426138 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426143 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426145 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426148 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426151 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.426996 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427004 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427009 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427014 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427018 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427021 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427024 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427028 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427031 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427034 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427036 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427039 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427042 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:21:54.428325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427044 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427047 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427049 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427052 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427055 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427057 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427060 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427062 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427065 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427069 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427073 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427075 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427079 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427081 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427084 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427087 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427090 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427092 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427095 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427098 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:21:54.428819 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427101 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427104 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427107 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427127 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427130 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427133 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427136 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427140 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427142 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427145 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427148 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427150 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427153 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427155 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427158 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427160 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427163 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427166 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427168 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:21:54.429308 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427171 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427174 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427176 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427179 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427181 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427184 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427186 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427189 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427191 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427194 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427203 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427207 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427211 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427213 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427216 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427219 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427222 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427225 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427228 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427230 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:21:54.429766 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427233 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427237 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427241 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427243 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427246 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427248 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427251 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427254 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427258 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427262 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427265 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427268 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427270 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427274 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427276 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427279 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427281 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427284 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427287 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:21:54.430249 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427290 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427292 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427295 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427298 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427301 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427303 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427305 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427308 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427311 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427313 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427316 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427318 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427321 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427324 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427326 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427852 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427858 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427861 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427864 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427867 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:21:54.430694 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427869 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427872 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427875 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427877 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427880 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427884 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427887 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427889 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427891 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427894 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427897 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427900 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427903 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427906 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427908 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427911 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427913 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427916 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427919 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:21:54.431270 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427921 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427924 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427927 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427929 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427932 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427935 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427937 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427940 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427943 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427945 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427948 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427951 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427954 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427956 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427960 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427964 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427968 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427971 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427975 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:21:54.431725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427978 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427981 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427984 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427987 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427989 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427992 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427995 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.427997 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428000 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428002 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428005 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428007 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428010 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428012 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428015 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428017 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428020 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428022 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428025 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428027 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:21:54.432238 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428030 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428032 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428035 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428037 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428040 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428042 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428045 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428047 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428050 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428052 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428055 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428060 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428062 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428066 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428069 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428072 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428074 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428077 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428079 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428082 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:21:54.432715 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428084 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428087 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428090 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428092 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428095 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428097 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428100 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428103 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428105 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428107 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428125 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428131 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428135 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428138 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428141 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428144 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428147 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428150 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428152 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:21:54.433252 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428156 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428159 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428162 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428164 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428263 2576 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428270 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428279 2576 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428284 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428288 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428292 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428296 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428300 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428304 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428307 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428311 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428314 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428317 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428320 2576 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428324 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428327 2576 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428330 2576 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428333 2576 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428337 2576 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428340 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:21:54.433701 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428344 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428347 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428350 2576 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428354 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428358 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428362 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428365 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428368 2576 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428371 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428374 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428377 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428380 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428389 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428392 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428396 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428400 2576 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428403 2576 flags.go:64] FLAG: --enable-server="true" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428406 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428411 2576 flags.go:64] FLAG: --event-burst="100" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428414 2576 flags.go:64] FLAG: --event-qps="50" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428417 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428420 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428423 2576 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428427 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:21:54.434278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428431 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428434 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428437 2576 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428440 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428443 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428446 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428449 2576 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428452 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428455 2576 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428458 2576 flags.go:64] FLAG: --feature-gates="" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428463 2576 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428466 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428469 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428472 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428475 2576 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428479 2576 flags.go:64] FLAG: --help="false" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428482 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428485 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428488 2576 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428491 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428496 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428500 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428504 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:21:54.434874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428507 2576 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428511 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428513 2576 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428516 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428520 2576 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428522 2576 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428525 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428528 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428532 2576 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428535 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428538 2576 flags.go:64] FLAG: --lock-file="" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428540 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428543 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428546 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428552 2576 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428555 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428558 2576 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428561 2576 flags.go:64] FLAG: --logging-format="text" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428564 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428567 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428570 2576 flags.go:64] FLAG: --manifest-url="" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428573 2576 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428577 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428580 2576 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428585 2576 flags.go:64] FLAG: --max-pods="110" Nov 01 22:21:54.435452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428588 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428591 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428594 2576 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428597 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428602 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428605 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428609 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428616 2576 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428619 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428622 2576 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428625 2576 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428628 2576 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428631 2576 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428634 2576 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428637 2576 flags.go:64] FLAG: --port="10250" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428640 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428644 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0dbb0c5dd5ea56a1f" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428647 2576 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428650 2576 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428654 2576 flags.go:64] FLAG: --register-node="true" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428656 2576 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428663 2576 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428666 2576 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428669 2576 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428672 2576 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428676 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:21:54.436042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428680 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428683 2576 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428686 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428688 2576 flags.go:64] FLAG: --runonce="false" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428691 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428694 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428698 2576 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428701 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428704 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428707 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428712 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428715 2576 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428720 2576 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428723 2576 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428726 2576 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428728 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428732 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428735 2576 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428738 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428743 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428746 2576 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428749 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428753 2576 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428756 2576 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428759 2576 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:21:54.436709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428762 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428765 2576 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428768 2576 flags.go:64] FLAG: --v="2" Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428772 2576 flags.go:64] FLAG: --version="false" Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428776 2576 flags.go:64] FLAG: --vmodule="" Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428781 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.428784 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428922 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428926 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428929 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428931 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428934 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428937 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428940 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428943 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428946 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428948 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428952 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428955 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428958 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428961 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:21:54.437324 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428963 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428966 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428969 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428971 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428973 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428976 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428978 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428981 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428984 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428986 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428990 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428994 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428997 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.428999 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429002 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429004 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429007 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429010 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429012 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:21:54.437817 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429015 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429017 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429020 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429023 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429025 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429028 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429030 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429033 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429036 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429040 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429043 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429045 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429048 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429050 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429053 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429056 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429058 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429061 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429063 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:21:54.438285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429066 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429069 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429071 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429074 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429076 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429079 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429081 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429084 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429086 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429089 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429091 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429094 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429096 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429099 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429101 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429104 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429106 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429124 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429129 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:21:54.438732 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429132 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429134 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429138 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429142 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429145 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429147 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429150 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429152 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429155 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429157 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429160 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429162 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429165 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429167 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429170 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429172 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429175 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429178 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429180 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429182 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:21:54.439193 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429185 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429187 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429190 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429192 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429195 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429199 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429202 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429205 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429208 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429211 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429213 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429216 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429218 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429221 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429224 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.429252 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:21:54.439685 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429295 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429299 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429303 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429305 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429308 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429311 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429314 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429316 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429319 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429322 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429325 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429328 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429330 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429333 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429337 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429341 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429344 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429347 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429350 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429353 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:21:54.440069 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429355 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429358 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429362 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429364 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429367 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429370 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429372 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429375 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429378 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429380 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429383 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429387 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429390 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429393 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429396 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429399 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429401 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429404 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429407 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:21:54.440537 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429409 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429412 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429414 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429417 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429419 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429422 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429424 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429428 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429431 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429433 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429436 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429438 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429441 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429443 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429446 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429450 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429452 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429455 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429458 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:21:54.440995 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429460 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429463 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429466 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429468 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429471 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429475 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429477 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429480 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429483 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429485 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429488 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429491 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429494 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429496 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429499 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429501 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429504 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429506 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429509 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:21:54.441663 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429511 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429514 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429516 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429519 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429522 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429524 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429527 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429530 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429533 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429538 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429541 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429545 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429547 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429550 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429553 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429555 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429558 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429561 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429564 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429567 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:21:54.442356 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429570 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429573 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429575 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429578 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429581 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429583 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429586 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429588 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.429592 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.436970 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.436986 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.438519 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.438527 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438585 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438590 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438594 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438597 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438600 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438603 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438606 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:21:54.442831 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438609 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438611 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438614 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438617 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438619 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438622 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438625 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438628 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438630 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438633 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438636 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438638 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438641 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438643 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438646 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438648 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438651 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438654 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438656 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438659 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:21:54.443325 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438661 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438664 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438668 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438671 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438674 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438676 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438679 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438681 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438684 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438687 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438689 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438692 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438694 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438698 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438702 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438705 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438708 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438711 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:21:54.443792 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438713 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438716 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438719 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438721 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438724 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438726 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438729 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438732 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438734 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438737 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438740 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438742 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438745 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438748 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438751 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438753 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438757 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438761 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438764 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:21:54.444362 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438767 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438770 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438772 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438775 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438778 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438780 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438783 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438785 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438788 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438790 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438793 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438795 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438798 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438800 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438803 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438805 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438808 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438810 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438813 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438815 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:21:54.444801 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438818 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438820 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438823 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438825 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438828 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438830 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438833 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438835 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438838 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438842 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438844 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438847 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438850 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438852 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438855 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438858 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438860 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438863 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438865 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438868 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:21:54.445285 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438871 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.438873 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.438902 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439042 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439047 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439050 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439053 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439056 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439058 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439061 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439064 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439067 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439069 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439072 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439075 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439078 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:21:54.445739 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439080 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439083 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439085 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439088 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439091 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439093 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439096 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439098 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439101 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439104 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439107 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439122 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439126 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439129 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439131 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439135 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439140 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439143 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439146 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:21:54.446122 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439149 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439152 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439154 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439157 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439159 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439162 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439164 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439167 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439169 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439172 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439175 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439177 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439180 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439183 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439185 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439188 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439190 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439193 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439196 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439205 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:21:54.446587 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439208 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439210 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439214 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439217 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439220 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439222 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439225 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439228 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439230 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439233 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439235 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439238 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439240 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439243 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439245 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439248 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439250 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439253 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439255 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439259 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:21:54.447045 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439261 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439264 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439267 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439270 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439272 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439275 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439277 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439280 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439283 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439285 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439288 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439290 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439293 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439295 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439298 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439301 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439303 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439306 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439309 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:21:54.447547 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439311 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439314 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439316 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439318 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439321 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439323 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439326 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439330 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439333 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439336 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439339 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439341 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439343 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439346 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439351 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:21:54.447989 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.439375 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439405 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439409 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439412 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439414 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439417 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439420 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439423 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439425 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439428 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439430 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439433 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439435 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439438 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439441 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439444 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439446 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439449 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439451 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439454 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:21:54.448360 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439456 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439459 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439461 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439463 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439466 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439468 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439471 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439473 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439476 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439478 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439481 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439484 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439488 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439490 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439493 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439495 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439499 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439503 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439507 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:21:54.448846 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439509 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439512 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439515 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439518 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439520 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439523 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439525 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439528 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439531 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439533 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439536 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439539 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439541 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439544 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439547 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439549 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439552 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439554 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439557 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439559 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:21:54.449322 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439562 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439564 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439567 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439570 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439573 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439577 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439580 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439583 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439586 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439588 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439591 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439593 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439596 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439598 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439601 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439603 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439606 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439608 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439611 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439613 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:21:54.449782 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439616 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439619 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439622 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439624 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439627 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439629 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439632 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439634 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439637 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439640 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439642 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439645 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439647 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439650 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439653 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439655 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439658 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439662 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439665 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:21:54.450427 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439667 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439670 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439672 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439675 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439677 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439680 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439682 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439685 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:54.439689 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.439743 2576 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.443598 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.444662 2576 server.go:1014] "Starting client certificate rotation" Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.444758 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.445578 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.446895 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:21:54.450866 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.449142 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:21:54.461068 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.461049 2576 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:21:54.464966 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.464950 2576 log.go:25] "Validated CRI v1 image API" Nov 01 22:21:54.465770 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.465749 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:21:54.469339 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.469316 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81e7547f-31d2-4e8c-b370-8ede5a4c5717:/dev/nvme0n1p4 ad8bed32-968f-4a0e-8f3b-236d5f925ee2:/dev/nvme0n1p3] Nov 01 22:21:54.469393 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.469340 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 22:21:54.474452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.474437 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:21:54.475746 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.475633 2576 manager.go:217] Machine: {Timestamp:2026-11-01 22:21:54.474080141 +0000 UTC m=+0.376115971 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099591 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d31d32504a7f62f2aca2ab1e08a5 SystemUUID:ec27d31d-3250-4a7f-62f2-aca2ab1e08a5 BootID:ab64b12d-0c75-484b-9df7-ef4571dc24a3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cf:01:92:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cf:01:92:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:8c:0d:ea:d8:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:21:54.476399 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.476388 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:21:54.476503 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.476491 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:21:54.479916 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.479893 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:21:54.480059 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.479918 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-55.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:21:54.480810 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.480801 2576 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:21:54.480844 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.480813 2576 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:21:54.480844 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.480838 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:21:54.480895 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.480852 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:21:54.482151 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.482141 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:21:54.486738 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.486727 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:21:54.486868 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.486859 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:21:54.486909 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.486901 2576 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:21:54.486939 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.486914 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:21:54.486969 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.486946 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:21:54.486969 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.486955 2576 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:21:54.486969 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.486962 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:21:54.487560 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.487545 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:21:54.491447 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.491421 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:21:54.492382 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.492364 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnx88" Nov 01 22:21:54.498746 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.498733 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:21:54.499478 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.499462 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnx88" Nov 01 22:21:54.499665 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.499652 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:21:54.499703 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.499683 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:21:54.499853 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.499843 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:21:54.499882 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.499862 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:21:54.499882 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.499872 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:21:54.499882 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.499880 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:21:54.499961 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.499890 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:21:54.499961 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.499899 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:21:54.500652 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.500638 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:21:54.500691 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.500655 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:21:54.500691 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.500666 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:21:54.500691 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.500675 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:21:54.500835 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.500700 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:21:54.500835 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.500714 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:21:54.501421 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.501409 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:21:54.501452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.501424 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:21:54.505190 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.505169 2576 server.go:1263] "Started kubelet" Nov 01 22:21:54.505284 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.505236 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:21:54.505792 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.505736 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:21:54.505882 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.505811 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:21:54.505962 ip-10-0-128-55 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:21:54.507866 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.507852 2576 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:21:54.508142 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.508128 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:21:54.511369 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.511350 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:21:54.512881 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.512852 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-128-55.us-east-2.compute.internal" not found Nov 01 22:21:54.513514 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.513494 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:21:54.513939 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:54.513922 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 22:21:54.514262 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.514248 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:21:54.514329 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.514262 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:21:54.514377 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:54.514334 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-55.us-east-2.compute.internal\" not found" Nov 01 22:21:54.515016 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.514999 2576 factory.go:55] Registering systemd factory Nov 01 22:21:54.515090 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515019 2576 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:21:54.515090 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515010 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:21:54.515090 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515032 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:21:54.515090 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515006 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:21:54.515298 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515147 2576 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:21:54.515298 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515156 2576 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:21:54.515298 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515269 2576 factory.go:153] Registering CRI-O factory Nov 01 22:21:54.515298 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515282 2576 factory.go:223] Registration of the crio container factory successfully Nov 01 22:21:54.515452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515349 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:21:54.515452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515371 2576 factory.go:103] Registering Raw factory Nov 01 22:21:54.515452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.515385 2576 manager.go:1199] Started watching for new ooms in manager Nov 01 22:21:54.516997 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.516981 2576 manager.go:319] Starting recovery of all containers Nov 01 22:21:54.519760 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.519738 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:21:54.525310 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:54.525284 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-55.us-east-2.compute.internal\" not found" node="ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.525745 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.525727 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-128-55.us-east-2.compute.internal" not found Nov 01 22:21:54.529087 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.529069 2576 manager.go:324] Recovery completed Nov 01 22:21:54.530409 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:54.530387 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 01 22:21:54.533638 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.533622 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:21:54.537900 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.537882 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:21:54.537984 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.537914 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:21:54.537984 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.537928 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:21:54.538583 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.538570 2576 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:21:54.538583 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.538579 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:21:54.538715 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.538593 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:21:54.538715 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.538598 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:21:54.539449 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.539431 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:21:54.539507 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.539467 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:21:54.539507 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.539484 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:21:54.541027 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.541014 2576 policy_none.go:50] "Start" Nov 01 22:21:54.541066 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.541040 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:21:54.541066 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.541055 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:21:54.543193 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.543182 2576 policy_none.go:44] "Start" Nov 01 22:21:54.582667 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.582635 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:21:54.583204 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.583178 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-128-55.us-east-2.compute.internal" not found Nov 01 22:21:54.585454 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.585441 2576 manager.go:341] "Starting Device Plugin manager" Nov 01 22:21:54.586245 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:54.586224 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:21:54.586245 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.586248 2576 server.go:87] "Starting device plugin registration server" Nov 01 22:21:54.586532 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.586517 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:21:54.586582 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.586538 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:21:54.586660 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.586644 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:21:54.586741 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.586723 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:21:54.586741 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.586736 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:21:54.587321 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:54.587304 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:21:54.587428 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:54.587344 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-55.us-east-2.compute.internal\" not found" Nov 01 22:21:54.648554 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.648520 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:21:54.650561 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.649741 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:21:54.650561 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.649764 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:21:54.650561 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.649813 2576 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:21:54.650561 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:54.649856 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 22:21:54.653352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.653336 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:21:54.687544 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.687481 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:21:54.688376 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.688362 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:21:54.688429 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.688390 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:21:54.688429 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.688402 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:21:54.688429 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.688428 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.695684 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.695542 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.750225 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.750194 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal"] Nov 01 22:21:54.752586 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.752570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.754338 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.754320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.786588 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.786569 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.789029 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.789015 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.796830 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.796816 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:21:54.799761 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.799748 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:21:54.916106 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.916077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ed9d7a120c3abfac5e4283e23adc9e4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"6ed9d7a120c3abfac5e4283e23adc9e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.916106 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.916106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:54.916309 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:54.916145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:55.016657 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.016583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ed9d7a120c3abfac5e4283e23adc9e4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"6ed9d7a120c3abfac5e4283e23adc9e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:55.016657 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.016621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:55.016657 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.016639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:55.016818 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.016701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:55.016818 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.016704 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:55.016818 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.016755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ed9d7a120c3abfac5e4283e23adc9e4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"6ed9d7a120c3abfac5e4283e23adc9e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:55.101042 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.101018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:55.103176 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.103158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:21:55.445280 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.445199 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 22:21:55.445780 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.445371 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 22:21:55.445780 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.445393 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 22:21:55.487657 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.487634 2576 apiserver.go:52] "Watching apiserver" Nov 01 22:21:55.494035 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.494013 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:21:55.494343 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.494325 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-9pqxl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-bqwkz","kube-system/global-pull-secret-syncer-cgcvk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv","openshift-multus/multus-8rwzp","openshift-multus/multus-additional-cni-plugins-s86mt","openshift-network-diagnostics/network-check-target-h5fn2","openshift-network-operator/iptables-alerter-pchz7","kube-system/konnectivity-agent-prbmb","kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-mjw8q","openshift-cluster-node-tuning-operator/tuned-zlskb","openshift-image-registry/node-ca-wbfqp"] Nov 01 22:21:55.496988 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.496968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.499268 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.499251 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.500656 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.500632 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:21:55.500656 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.500636 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:21:55.500800 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.500712 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-79k59\"" Nov 01 22:21:55.501571 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.501551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:55.501661 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.501616 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:21:55.502423 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.502405 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:21:55.502544 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.502429 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:21:55.502544 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.502440 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:21:55.502544 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.502501 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vvcmq\"" Nov 01 22:21:55.502720 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.502556 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:21:55.502720 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.502570 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:21:55.502720 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.502630 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:21:55.503808 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.503793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.505781 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.505740 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:16:55 +0000 UTC" deadline="2028-06-28 03:10:37.904019108 +0000 UTC" Nov 01 22:21:55.505781 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.505779 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14500h48m42.398242172s" Nov 01 22:21:55.506198 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.506169 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.508616 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.508600 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.510845 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.510827 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:21:55.510950 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.510893 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:21:55.513200 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.513167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.514396 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.514376 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 22:21:55.515614 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.515593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:21:55.519196 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.519165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:55.519285 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.519255 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:21:55.520385 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svt8j\" (UniqueName: \"kubernetes.io/projected/d35d0387-f298-43a2-977a-51386c17077a-kube-api-access-svt8j\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.520480 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-k8s-cni-cncf-io\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.520480 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-multus\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.520561 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-etc-kubernetes\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.520561 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-env-overrides\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.520628 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-device-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.520628 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-system-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.520628 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cedefae3-8d26-4f04-a6c1-78d62920c2d6-host-slash\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.520744 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gjs\" (UniqueName: \"kubernetes.io/projected/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-kube-api-access-q5gjs\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.520744 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:55.520744 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-netd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.520835 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-socket-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.520835 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-netns\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.520835 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cnibin\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.520958 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520857 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-ovn\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.520958 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-slash\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.520958 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520934 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-etc-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521079 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.520968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-log-socket\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521079 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.521197 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521197 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-script-lib\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521294 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-sys-fs\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.521294 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-bin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.521294 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-var-lib-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521559 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-bin\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521559 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d35d0387-f298-43a2-977a-51386c17077a-ovn-node-metrics-cert\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521559 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-system-cni-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.521559 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-hosts-file\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.521559 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521559 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.521559 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-kubelet-config\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:55.521559 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvfz\" (UniqueName: \"kubernetes.io/projected/cedefae3-8d26-4f04-a6c1-78d62920c2d6-kube-api-access-zpvfz\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-tmp-dir\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-kubelet\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ww2v\" (UniqueName: \"kubernetes.io/projected/0e214ead-da27-47f8-8196-34d885b7b4dc-kube-api-access-8ww2v\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb6rm\" (UniqueName: \"kubernetes.io/projected/77c13e2e-6095-4786-85d2-278dd6bf9f69-kube-api-access-bb6rm\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-systemd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-cnibin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-cni-binary-copy\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-kubelet\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.521867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnvgl\" (UniqueName: \"kubernetes.io/projected/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-kube-api-access-mnvgl\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eba18759-4d3b-468a-9a93-0044457a0b93-agent-certs\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.521975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eba18759-4d3b-468a-9a93-0044457a0b93-konnectivity-ca\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-socket-dir-parent\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-node-log\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-registration-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-hostroot\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-multus-certs\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-netns\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-config\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-dbus\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:55.522363 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-systemd-units\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.522899 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.522899 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-conf-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.522899 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-daemon-config\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.522899 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.522464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cedefae3-8d26-4f04-a6c1-78d62920c2d6-iptables-alerter-script\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.524032 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.524015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.544296 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.544273 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:21:55.544392 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.544341 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kztjb" Nov 01 22:21:55.551364 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.551344 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kztjb" Nov 01 22:21:55.609894 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.609874 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:21:55.610009 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.609918 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:21:55.611205 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.610346 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:21:55.611205 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.610432 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:21:55.611205 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.610433 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qjk9v\"" Nov 01 22:21:55.611205 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.610726 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:21:55.611205 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.610799 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:21:55.611205 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.610805 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:21:55.611205 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.610847 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:21:55.611205 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.610864 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:21:55.611205 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.610970 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vv5lb\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611268 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dclh4\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611482 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611505 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611513 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p26bd\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611522 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zwl5x\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611527 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hpp8\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611537 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611547 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611561 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611623 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611655 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:21:55.611721 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611660 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:21:55.612243 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611799 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:21:55.612243 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611823 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j5zd8\"" Nov 01 22:21:55.612243 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.611860 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:21:55.615712 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.615694 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:21:55.622626 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.622608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-socket-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.622775 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.622633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-serviceca\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.622841 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.622799 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-socket-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.623053 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-lib-modules\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.623092 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-netns\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.623092 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cnibin\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.623092 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-tuned\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.623264 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-ovn\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623264 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwdz\" (UniqueName: \"kubernetes.io/projected/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-kube-api-access-tdwdz\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.623264 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-tmp\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.623264 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623174 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-netns\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.623264 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623210 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cnibin\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.623264 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623237 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-ovn\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623264 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-slash\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-etc-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-log-socket\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-slash\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623340 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-etc-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623374 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-log-socket\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-script-lib\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-sys-fs\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-bin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-conf\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623508 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623531 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-sys-fs\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623549 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-bin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.623573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-var-lib-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-bin\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623601 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-var-lib-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d35d0387-f298-43a2-977a-51386c17077a-ovn-node-metrics-cert\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.623990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-system-cni-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-hosts-file\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624043 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-bin\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-host\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624108 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-system-cni-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-run\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-hosts-file\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-var-lib-kubelet\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624324 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-kubelet-config\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:55.624352 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624323 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:21:55.625096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624413 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-script-lib\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.625096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysconfig\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.625096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624554 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-kubelet-config\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:55.625096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-host\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.625096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624679 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.625096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9q8n\" (UniqueName: \"kubernetes.io/projected/665f24e2-f8fb-4bc7-8958-191e776f78fd-kube-api-access-p9q8n\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.625096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.625096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvfz\" (UniqueName: \"kubernetes.io/projected/cedefae3-8d26-4f04-a6c1-78d62920c2d6-kube-api-access-zpvfz\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.625096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.624913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-tmp-dir\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.625560 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-sys\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.625560 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-kubelet\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.625560 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625460 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.625560 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ww2v\" (UniqueName: \"kubernetes.io/projected/0e214ead-da27-47f8-8196-34d885b7b4dc-kube-api-access-8ww2v\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.625784 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6rm\" (UniqueName: \"kubernetes.io/projected/77c13e2e-6095-4786-85d2-278dd6bf9f69-kube-api-access-bb6rm\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.625784 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.625784 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-systemd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.625784 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-cnibin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.625784 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-cni-binary-copy\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.625784 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-kubelet\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.625784 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.626158 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625573 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-kubelet\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.626158 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625945 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-systemd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.626158 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvgl\" (UniqueName: \"kubernetes.io/projected/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-kube-api-access-mnvgl\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.626158 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.626010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eba18759-4d3b-468a-9a93-0044457a0b93-agent-certs\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:21:55.626158 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.626043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eba18759-4d3b-468a-9a93-0044457a0b93-konnectivity-ca\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:21:55.626158 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.626053 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-cnibin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.626158 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.626077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-socket-dir-parent\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.626548 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.626213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:55.626548 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.626245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tspch\" (UniqueName: \"kubernetes.io/projected/7c8ec790-a230-403d-8972-5634c7d2ae0f-kube-api-access-tspch\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:55.626548 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.626271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-kubelet\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.626548 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.626280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-kubernetes\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627064 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-systemd\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627143 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-cni-binary-copy\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-node-log\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-registration-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-hostroot\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627342 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627396 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eba18759-4d3b-468a-9a93-0044457a0b93-konnectivity-ca\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-multus-certs\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.628149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.625855 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-tmp-dir\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.628738 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627509 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-hostroot\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.628738 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627543 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-node-log\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.628738 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627593 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-registration-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.628738 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627636 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-multus-certs\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.628738 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-modprobe-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-netns\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-config\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-dbus\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-systemd-units\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.629007 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.628995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-conf-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-daemon-config\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cedefae3-8d26-4f04-a6c1-78d62920c2d6-iptables-alerter-script\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svt8j\" (UniqueName: \"kubernetes.io/projected/d35d0387-f298-43a2-977a-51386c17077a-kube-api-access-svt8j\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-k8s-cni-cncf-io\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627754 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.627708 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-socket-dir-parent\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-multus\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-etc-kubernetes\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-env-overrides\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629265 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-device-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-system-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cedefae3-8d26-4f04-a6c1-78d62920c2d6-host-slash\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gjs\" (UniqueName: \"kubernetes.io/projected/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-kube-api-access-q5gjs\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.629843 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-daemon-config\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.630963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.630165 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d35d0387-f298-43a2-977a-51386c17077a-ovn-node-metrics-cert\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.632026 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.630966 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-systemd-units\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.632026 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.631059 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-etc-kubernetes\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.632026 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.631140 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.632026 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.631207 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-k8s-cni-cncf-io\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.632026 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.631214 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.632026 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.631298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-system-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.632388 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.632023 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-config\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.632388 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.632218 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eba18759-4d3b-468a-9a93-0044457a0b93-agent-certs\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:21:55.632590 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.632566 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cedefae3-8d26-4f04-a6c1-78d62920c2d6-host-slash\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.632652 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.632636 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-conf-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.633359 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.632763 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:21:55.633359 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.632833 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cedefae3-8d26-4f04-a6c1-78d62920c2d6-iptables-alerter-script\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.633359 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.632876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-netd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.633359 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.632989 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret podName:330ca60e-5fa1-4a41-8617-73450d13cac2 nodeName:}" failed. No retries permitted until 2026-11-01 22:21:56.132946223 +0000 UTC m=+2.034982060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret") pod "global-pull-secret-syncer-cgcvk" (UID: "330ca60e-5fa1-4a41-8617-73450d13cac2") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:21:55.633359 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.633114 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-dbus\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:55.633359 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.633160 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-env-overrides\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.633359 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.633205 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-multus\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.633359 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.633041 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-netd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.633359 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.633298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-device-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.633914 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.633548 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:21:55.633914 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.633566 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:21:55.633914 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.633593 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gwb7v for pod openshift-network-diagnostics/network-check-target-h5fn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:21:55.633914 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.633658 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v podName:5029000f-c7ae-4f6f-9b0c-5240e385f697 nodeName:}" failed. No retries permitted until 2026-11-01 22:21:56.133641602 +0000 UTC m=+2.035677443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gwb7v" (UniqueName: "kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v") pod "network-check-target-h5fn2" (UID: "5029000f-c7ae-4f6f-9b0c-5240e385f697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:21:55.635391 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.635364 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvgl\" (UniqueName: \"kubernetes.io/projected/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-kube-api-access-mnvgl\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.635660 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.635637 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6rm\" (UniqueName: \"kubernetes.io/projected/77c13e2e-6095-4786-85d2-278dd6bf9f69-kube-api-access-bb6rm\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.635660 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.635656 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvfz\" (UniqueName: \"kubernetes.io/projected/cedefae3-8d26-4f04-a6c1-78d62920c2d6-kube-api-access-zpvfz\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.636551 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.636212 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-netns\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.636692 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.636670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ww2v\" (UniqueName: \"kubernetes.io/projected/0e214ead-da27-47f8-8196-34d885b7b4dc-kube-api-access-8ww2v\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.640866 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.640834 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svt8j\" (UniqueName: \"kubernetes.io/projected/d35d0387-f298-43a2-977a-51386c17077a-kube-api-access-svt8j\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.643285 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.643259 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gjs\" (UniqueName: \"kubernetes.io/projected/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-kube-api-access-q5gjs\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.733473 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-tuned\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.733473 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwdz\" (UniqueName: \"kubernetes.io/projected/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-kube-api-access-tdwdz\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.733473 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-tmp\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.733746 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-conf\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.733746 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-host\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.733746 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733652 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-conf\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.733746 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-host\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.733746 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733716 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-run\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.733746 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-run\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.733746 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-var-lib-kubelet\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysconfig\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-host\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733813 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-var-lib-kubelet\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9q8n\" (UniqueName: \"kubernetes.io/projected/665f24e2-f8fb-4bc7-8958-191e776f78fd-kube-api-access-p9q8n\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733867 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-host\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733879 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysconfig\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-sys\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733910 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-sys\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.733995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tspch\" (UniqueName: \"kubernetes.io/projected/7c8ec790-a230-403d-8972-5634c7d2ae0f-kube-api-access-tspch\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-kubernetes\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-systemd\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-modprobe-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734103 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.734108 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.734189 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs podName:7c8ec790-a230-403d-8972-5634c7d2ae0f nodeName:}" failed. No retries permitted until 2026-11-01 22:21:56.234148082 +0000 UTC m=+2.136183916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs") pod "network-metrics-daemon-mjw8q" (UID: "7c8ec790-a230-403d-8972-5634c7d2ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734221 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734267 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-kubernetes\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734292 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-systemd\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-serviceca\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-lib-modules\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-modprobe-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-lib-modules\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.734826 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.734822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-serviceca\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.735795 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.735776 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-tmp\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.735878 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.735861 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-tuned\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.742409 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.742388 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwdz\" (UniqueName: \"kubernetes.io/projected/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-kube-api-access-tdwdz\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.742521 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.742503 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9q8n\" (UniqueName: \"kubernetes.io/projected/665f24e2-f8fb-4bc7-8958-191e776f78fd-kube-api-access-p9q8n\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.742617 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.742599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tspch\" (UniqueName: \"kubernetes.io/projected/7c8ec790-a230-403d-8972-5634c7d2ae0f-kube-api-access-tspch\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:55.781891 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.781853 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba2bbee94e76dfc255e8cf94df97bf1.slice/crio-06ea91c2eb041d2cb1012d947bb6e87ac42c33c04f87ef3a1a3add0b88ee20e2 WatchSource:0}: Error finding container 06ea91c2eb041d2cb1012d947bb6e87ac42c33c04f87ef3a1a3add0b88ee20e2: Status 404 returned error can't find the container with id 06ea91c2eb041d2cb1012d947bb6e87ac42c33c04f87ef3a1a3add0b88ee20e2 Nov 01 22:21:55.782309 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.782291 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed9d7a120c3abfac5e4283e23adc9e4.slice/crio-939fd86d5f22e133516813f2ad8bca2b29b61dd22b0c5123d517a095b8594251 WatchSource:0}: Error finding container 939fd86d5f22e133516813f2ad8bca2b29b61dd22b0c5123d517a095b8594251: Status 404 returned error can't find the container with id 939fd86d5f22e133516813f2ad8bca2b29b61dd22b0c5123d517a095b8594251 Nov 01 22:21:55.787406 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.787387 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:21:55.822516 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.822487 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:21:55.832276 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.832254 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5265e1c6_2a08_4439_b6a9_42ed2fdbd81a.slice/crio-7be64e7cb04ae2732b3cdaf6102ec67ccef684c704d58f4dbc68e64247f5d05a WatchSource:0}: Error finding container 7be64e7cb04ae2732b3cdaf6102ec67ccef684c704d58f4dbc68e64247f5d05a: Status 404 returned error can't find the container with id 7be64e7cb04ae2732b3cdaf6102ec67ccef684c704d58f4dbc68e64247f5d05a Nov 01 22:21:55.859242 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.859220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:21:55.866885 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.866865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:21:55.868916 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.868893 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35d0387_f298_43a2_977a_51386c17077a.slice/crio-ad97dc80f450d2d49b58649a2bf82779c108a3f04b7e0a80ddfda052eb7c9c46 WatchSource:0}: Error finding container ad97dc80f450d2d49b58649a2bf82779c108a3f04b7e0a80ddfda052eb7c9c46: Status 404 returned error can't find the container with id ad97dc80f450d2d49b58649a2bf82779c108a3f04b7e0a80ddfda052eb7c9c46 Nov 01 22:21:55.875698 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.875671 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e214ead_da27_47f8_8196_34d885b7b4dc.slice/crio-33099fa3a95320dbd3c785669058a2167d8d08283e0faded4365ac9df5c9b024 WatchSource:0}: Error finding container 33099fa3a95320dbd3c785669058a2167d8d08283e0faded4365ac9df5c9b024: Status 404 returned error can't find the container with id 33099fa3a95320dbd3c785669058a2167d8d08283e0faded4365ac9df5c9b024 Nov 01 22:21:55.877708 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.877681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8rwzp" Nov 01 22:21:55.886553 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.886532 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c13e2e_6095_4786_85d2_278dd6bf9f69.slice/crio-e7ff964982721b9723e2328e469703a85263e70d75cc0db3f88f28d3040f395a WatchSource:0}: Error finding container e7ff964982721b9723e2328e469703a85263e70d75cc0db3f88f28d3040f395a: Status 404 returned error can't find the container with id e7ff964982721b9723e2328e469703a85263e70d75cc0db3f88f28d3040f395a Nov 01 22:21:55.911629 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.911608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:21:55.914353 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.914332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:21:55.919528 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.919510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:21:55.921230 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.921212 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c99e3c_387a_4e15_bb7c_4dc72c91a2c0.slice/crio-b0c2d46712be5b919c1e1bd150ff6236ce1c491511ac8b91ff046f63c75398fb WatchSource:0}: Error finding container b0c2d46712be5b919c1e1bd150ff6236ce1c491511ac8b91ff046f63c75398fb: Status 404 returned error can't find the container with id b0c2d46712be5b919c1e1bd150ff6236ce1c491511ac8b91ff046f63c75398fb Nov 01 22:21:55.924004 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.923957 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedefae3_8d26_4f04_a6c1_78d62920c2d6.slice/crio-f435507ec658d780a06e7ad112fbf8f1ca1bf7380f79f5e54ce636a7407ec338 WatchSource:0}: Error finding container f435507ec658d780a06e7ad112fbf8f1ca1bf7380f79f5e54ce636a7407ec338: Status 404 returned error can't find the container with id f435507ec658d780a06e7ad112fbf8f1ca1bf7380f79f5e54ce636a7407ec338 Nov 01 22:21:55.929462 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.929443 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba18759_4d3b_468a_9a93_0044457a0b93.slice/crio-e5732415a7e4a063b587ad384d31a8bfb5d827c29650fcc73aa8a6bde1d09d18 WatchSource:0}: Error finding container e5732415a7e4a063b587ad384d31a8bfb5d827c29650fcc73aa8a6bde1d09d18: Status 404 returned error can't find the container with id e5732415a7e4a063b587ad384d31a8bfb5d827c29650fcc73aa8a6bde1d09d18 Nov 01 22:21:55.930474 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.930455 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:21:55.941101 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.941081 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665f24e2_f8fb_4bc7_8958_191e776f78fd.slice/crio-b48fe713965e5df1699ee63213b1f31f320d21a319e7dd963744a500d1676710 WatchSource:0}: Error finding container b48fe713965e5df1699ee63213b1f31f320d21a319e7dd963744a500d1676710: Status 404 returned error can't find the container with id b48fe713965e5df1699ee63213b1f31f320d21a319e7dd963744a500d1676710 Nov 01 22:21:55.955910 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:55.955884 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:21:55.965506 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:21:55.965483 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc57449_d2a8_41fa_808b_6ef9a3517bc6.slice/crio-f1608664a2958c899b08cabcc1486633d47b6a1cca5fd8000b7bbc447972f433 WatchSource:0}: Error finding container f1608664a2958c899b08cabcc1486633d47b6a1cca5fd8000b7bbc447972f433: Status 404 returned error can't find the container with id f1608664a2958c899b08cabcc1486633d47b6a1cca5fd8000b7bbc447972f433 Nov 01 22:21:55.967460 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.967442 2576 kuberuntime_manager.go:1707] "Unhandled Error" err="container node-ca start failed in pod node-ca-wbfqp_openshift-image-registry(cdc57449-d2a8-41fa-808b-6ef9a3517bc6): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Nov 01 22:21:55.967524 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:55.967486 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-image-registry/node-ca-wbfqp" podUID="cdc57449-d2a8-41fa-808b-6ef9a3517bc6" Nov 01 22:21:56.137218 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.137118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:21:56.137218 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.137174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:56.137422 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.137284 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:21:56.137422 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.137308 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:21:56.137422 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.137339 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:21:56.137422 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.137355 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gwb7v for pod openshift-network-diagnostics/network-check-target-h5fn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:21:56.137422 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.137340 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret podName:330ca60e-5fa1-4a41-8617-73450d13cac2 nodeName:}" failed. No retries permitted until 2026-11-01 22:21:57.137325701 +0000 UTC m=+3.039361521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret") pod "global-pull-secret-syncer-cgcvk" (UID: "330ca60e-5fa1-4a41-8617-73450d13cac2") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:21:56.137593 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.137428 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v podName:5029000f-c7ae-4f6f-9b0c-5240e385f697 nodeName:}" failed. No retries permitted until 2026-11-01 22:21:57.137410076 +0000 UTC m=+3.039445897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwb7v" (UniqueName: "kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v") pod "network-check-target-h5fn2" (UID: "5029000f-c7ae-4f6f-9b0c-5240e385f697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:21:56.238443 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.238412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:56.238615 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.238571 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:21:56.238671 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.238626 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs podName:7c8ec790-a230-403d-8972-5634c7d2ae0f nodeName:}" failed. No retries permitted until 2026-11-01 22:21:57.238611854 +0000 UTC m=+3.140647676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs") pod "network-metrics-daemon-mjw8q" (UID: "7c8ec790-a230-403d-8972-5634c7d2ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:21:56.519542 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.519436 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:21:56.554372 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.554259 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:16:56 +0000 UTC" deadline="2028-05-14 13:50:25.335501738 +0000 UTC" Nov 01 22:21:56.554372 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.554305 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13431h28m28.781201925s" Nov 01 22:21:56.639785 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.639729 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:21:56.653770 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.653202 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:56.653770 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.653335 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:21:56.689001 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.688363 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wbfqp" event={"ID":"cdc57449-d2a8-41fa-808b-6ef9a3517bc6","Type":"ContainerStarted","Data":"f1608664a2958c899b08cabcc1486633d47b6a1cca5fd8000b7bbc447972f433"} Nov 01 22:21:56.695010 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.694957 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zlskb" event={"ID":"665f24e2-f8fb-4bc7-8958-191e776f78fd","Type":"ContainerStarted","Data":"b48fe713965e5df1699ee63213b1f31f320d21a319e7dd963744a500d1676710"} Nov 01 22:21:56.697471 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:56.697440 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11509ac76e128ae597c6ac16bf61ea639d70c1a8f19835bb5fc0d4c9fcf3867c\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-image-registry/node-ca-wbfqp" podUID="cdc57449-d2a8-41fa-808b-6ef9a3517bc6" Nov 01 22:21:56.700055 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.700028 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-prbmb" event={"ID":"eba18759-4d3b-468a-9a93-0044457a0b93","Type":"ContainerStarted","Data":"e5732415a7e4a063b587ad384d31a8bfb5d827c29650fcc73aa8a6bde1d09d18"} Nov 01 22:21:56.705385 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.705350 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pchz7" event={"ID":"cedefae3-8d26-4f04-a6c1-78d62920c2d6","Type":"ContainerStarted","Data":"f435507ec658d780a06e7ad112fbf8f1ca1bf7380f79f5e54ce636a7407ec338"} Nov 01 22:21:56.713806 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.713771 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86mt" event={"ID":"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0","Type":"ContainerStarted","Data":"b0c2d46712be5b919c1e1bd150ff6236ce1c491511ac8b91ff046f63c75398fb"} Nov 01 22:21:56.733613 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.732949 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerStarted","Data":"33099fa3a95320dbd3c785669058a2167d8d08283e0faded4365ac9df5c9b024"} Nov 01 22:21:56.736164 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.736132 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" event={"ID":"d35d0387-f298-43a2-977a-51386c17077a","Type":"ContainerStarted","Data":"ad97dc80f450d2d49b58649a2bf82779c108a3f04b7e0a80ddfda052eb7c9c46"} Nov 01 22:21:56.738496 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.738467 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"6ed9d7a120c3abfac5e4283e23adc9e4","Type":"ContainerStarted","Data":"939fd86d5f22e133516813f2ad8bca2b29b61dd22b0c5123d517a095b8594251"} Nov 01 22:21:56.744310 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.744281 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8rwzp" event={"ID":"77c13e2e-6095-4786-85d2-278dd6bf9f69","Type":"ContainerStarted","Data":"e7ff964982721b9723e2328e469703a85263e70d75cc0db3f88f28d3040f395a"} Nov 01 22:21:56.750183 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.750156 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9pqxl" event={"ID":"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a","Type":"ContainerStarted","Data":"7be64e7cb04ae2732b3cdaf6102ec67ccef684c704d58f4dbc68e64247f5d05a"} Nov 01 22:21:56.761094 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:56.761067 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"cba2bbee94e76dfc255e8cf94df97bf1","Type":"ContainerStarted","Data":"06ea91c2eb041d2cb1012d947bb6e87ac42c33c04f87ef3a1a3add0b88ee20e2"} Nov 01 22:21:57.147405 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:57.146701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:21:57.147405 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:57.146747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:57.147405 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.146872 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:21:57.147405 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.146926 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret podName:330ca60e-5fa1-4a41-8617-73450d13cac2 nodeName:}" failed. No retries permitted until 2026-11-01 22:21:59.146913691 +0000 UTC m=+5.048949511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret") pod "global-pull-secret-syncer-cgcvk" (UID: "330ca60e-5fa1-4a41-8617-73450d13cac2") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:21:57.147405 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.147306 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:21:57.147405 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.147320 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:21:57.147405 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.147330 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gwb7v for pod openshift-network-diagnostics/network-check-target-h5fn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:21:57.147405 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.147372 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v podName:5029000f-c7ae-4f6f-9b0c-5240e385f697 nodeName:}" failed. No retries permitted until 2026-11-01 22:21:59.147361258 +0000 UTC m=+5.049397077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwb7v" (UniqueName: "kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v") pod "network-check-target-h5fn2" (UID: "5029000f-c7ae-4f6f-9b0c-5240e385f697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:21:57.247400 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:57.247358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:57.247595 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.247561 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:21:57.247674 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.247631 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs podName:7c8ec790-a230-403d-8972-5634c7d2ae0f nodeName:}" failed. No retries permitted until 2026-11-01 22:21:59.247611518 +0000 UTC m=+5.149647340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs") pod "network-metrics-daemon-mjw8q" (UID: "7c8ec790-a230-403d-8972-5634c7d2ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:21:57.555528 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:57.555342 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:16:56 +0000 UTC" deadline="2028-05-01 04:22:06.705684891 +0000 UTC" Nov 01 22:21:57.555528 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:57.555441 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13110h0m9.150250328s" Nov 01 22:21:57.651374 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:57.650604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:21:57.651374 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.650739 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:21:57.651374 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:57.651157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:57.651374 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:57.651295 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:21:58.652899 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:58.652854 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:58.653434 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:58.652995 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:21:59.166605 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:59.166563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:21:59.166802 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:59.166672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:21:59.166878 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.166847 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:21:59.166878 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.166869 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:21:59.166987 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.166885 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gwb7v for pod openshift-network-diagnostics/network-check-target-h5fn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:21:59.166987 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.166976 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v podName:5029000f-c7ae-4f6f-9b0c-5240e385f697 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:03.16695775 +0000 UTC m=+9.068993573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwb7v" (UniqueName: "kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v") pod "network-check-target-h5fn2" (UID: "5029000f-c7ae-4f6f-9b0c-5240e385f697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:21:59.167156 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.167061 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:21:59.167156 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.167107 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret podName:330ca60e-5fa1-4a41-8617-73450d13cac2 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:03.167093324 +0000 UTC m=+9.069129161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret") pod "global-pull-secret-syncer-cgcvk" (UID: "330ca60e-5fa1-4a41-8617-73450d13cac2") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:21:59.267475 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:59.267354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:59.267646 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.267577 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:21:59.267708 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.267648 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs podName:7c8ec790-a230-403d-8972-5634c7d2ae0f nodeName:}" failed. No retries permitted until 2026-11-01 22:22:03.267628362 +0000 UTC m=+9.169664185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs") pod "network-metrics-daemon-mjw8q" (UID: "7c8ec790-a230-403d-8972-5634c7d2ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:21:59.651066 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:59.651036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:21:59.651230 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.651186 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:21:59.651893 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:59.651697 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:21:59.651893 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:21:59.651846 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:21:59.799998 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:21:59.799321 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86mt" event={"ID":"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0","Type":"ContainerStarted","Data":"18ba5b40d83b908087c7303cc3752bcd252747d18ded2b54095c6f37f4982edf"} Nov 01 22:22:00.653337 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:00.653255 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:00.653508 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:00.653395 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:00.808283 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:00.808171 2576 generic.go:356] "Generic (PLEG): container finished" podID="cba2bbee94e76dfc255e8cf94df97bf1" containerID="98e957fbf79758a8e0ac54014bb222a78b45ce361604ed410b4ca69fdf47eb7e" exitCode=0 Nov 01 22:22:00.808283 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:00.808243 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"cba2bbee94e76dfc255e8cf94df97bf1","Type":"ContainerDied","Data":"98e957fbf79758a8e0ac54014bb222a78b45ce361604ed410b4ca69fdf47eb7e"} Nov 01 22:22:00.817335 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:00.816756 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-prbmb" event={"ID":"eba18759-4d3b-468a-9a93-0044457a0b93","Type":"ContainerStarted","Data":"e14c67fcb1475f10709c7d3f50979d1b92f7810e603883c36fb0a22a47bbb539"} Nov 01 22:22:00.820283 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:00.819506 2576 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="18ba5b40d83b908087c7303cc3752bcd252747d18ded2b54095c6f37f4982edf" exitCode=0 Nov 01 22:22:00.820283 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:00.819539 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86mt" event={"ID":"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0","Type":"ContainerDied","Data":"18ba5b40d83b908087c7303cc3752bcd252747d18ded2b54095c6f37f4982edf"} Nov 01 22:22:01.394338 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:01.394253 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:22:01.395008 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:01.394983 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:22:01.410972 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:01.410916 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-prbmb" podStartSLOduration=2.071628997 podStartE2EDuration="6.410897712s" totalImagesPullingTime="4.339268715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:00.862099061 +0000 UTC m=+6.764134905" watchObservedRunningTime="2026-11-01 22:22:01.410897712 +0000 UTC m=+7.312933556" Nov 01 22:22:01.650535 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:01.650440 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:01.650695 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:01.650591 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:01.651029 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:01.651003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:01.651143 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:01.651113 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:01.826825 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:01.826093 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"6ed9d7a120c3abfac5e4283e23adc9e4","Type":"ContainerStarted","Data":"ff6eed3f0f435db8701501c02704fc4bf50ec06c64035a1cabda6ecb50b3dbcb"} Nov 01 22:22:01.840794 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:01.840677 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"cba2bbee94e76dfc255e8cf94df97bf1","Type":"ContainerStarted","Data":"53ad51a4a498b6197aa4ec3fef71a690de45cd67a346f96961ad60c5a5af0dcf"} Nov 01 22:22:01.841966 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:01.841794 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" podStartSLOduration=6.841773657 podStartE2EDuration="6.841773657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:01.841491656 +0000 UTC m=+7.743527504" watchObservedRunningTime="2026-11-01 22:22:01.841773657 +0000 UTC m=+7.743809502" Nov 01 22:22:02.652837 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:02.652303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:02.652837 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:02.652451 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:02.846222 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:02.846166 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerStarted","Data":"5a296f4606948844815c75184cb85207b6e661d2c918032f53b2709817520594"} Nov 01 22:22:02.846735 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:02.846705 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:03.204886 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:03.204838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:03.205053 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:03.204930 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:03.205127 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.205092 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:03.205127 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.205110 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:03.205127 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.205124 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gwb7v for pod openshift-network-diagnostics/network-check-target-h5fn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:03.205309 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.205205 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v podName:5029000f-c7ae-4f6f-9b0c-5240e385f697 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:11.205184631 +0000 UTC m=+17.107220468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwb7v" (UniqueName: "kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v") pod "network-check-target-h5fn2" (UID: "5029000f-c7ae-4f6f-9b0c-5240e385f697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:03.205650 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.205628 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:03.205714 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.205667 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret podName:330ca60e-5fa1-4a41-8617-73450d13cac2 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:11.205657467 +0000 UTC m=+17.107693301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret") pod "global-pull-secret-syncer-cgcvk" (UID: "330ca60e-5fa1-4a41-8617-73450d13cac2") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:03.306220 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:03.305727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:03.306220 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.305888 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:03.306220 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.305957 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs podName:7c8ec790-a230-403d-8972-5634c7d2ae0f nodeName:}" failed. No retries permitted until 2026-11-01 22:22:11.305938843 +0000 UTC m=+17.207974667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs") pod "network-metrics-daemon-mjw8q" (UID: "7c8ec790-a230-403d-8972-5634c7d2ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:03.651124 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:03.650570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:03.651124 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.650702 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:03.651124 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:03.650783 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:03.651124 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:03.650873 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:04.151072 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.151041 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:22:04.595725 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.595617 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:22:04.151058381Z","UUID":"653d7fb5-0516-41ec-8568-175894143560","Handler":null,"Name":"","Endpoint":""} Nov 01 22:22:04.597348 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.597322 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:22:04.597348 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.597351 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:22:04.656215 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.656183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:04.656410 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:04.656368 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:04.853733 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.853651 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerStarted","Data":"98dbc55a230ac12b17049c489e46a7a7ee745b344bd285d824b1a0acd2ff0024"} Nov 01 22:22:04.856908 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.856881 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8rwzp" event={"ID":"77c13e2e-6095-4786-85d2-278dd6bf9f69","Type":"ContainerStarted","Data":"39aa97d209f1a87fb415093858c5dab0fdffc28fe2343f9edd769e63ee559a4e"} Nov 01 22:22:04.858489 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.858462 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9pqxl" event={"ID":"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a","Type":"ContainerStarted","Data":"0f70dfe28e11eaf2577ab73196739aa64d1cc38ec2c911a4067cc42ba19655d0"} Nov 01 22:22:04.875718 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.875671 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" podStartSLOduration=9.875654681 podStartE2EDuration="9.875654681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:01.857561353 +0000 UTC m=+7.759597198" watchObservedRunningTime="2026-11-01 22:22:04.875654681 +0000 UTC m=+10.777690522" Nov 01 22:22:04.876181 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:04.876142 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8rwzp" podStartSLOduration=1.312835319 podStartE2EDuration="9.87612785s" totalImagesPullingTime="8.563292531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:04.875016544 +0000 UTC m=+10.777052407" watchObservedRunningTime="2026-11-01 22:22:04.87612785 +0000 UTC m=+10.778163688" Nov 01 22:22:05.650588 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:05.650380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:05.651018 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:05.650417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:05.651018 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:05.650702 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:05.651018 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:05.650774 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:05.863203 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:05.863165 2576 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="dd67ea480680b5f57832a9778364d69206aa53e221fc2eb19250f2f57809b3c8" exitCode=0 Nov 01 22:22:05.863368 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:05.863255 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86mt" event={"ID":"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0","Type":"ContainerDied","Data":"dd67ea480680b5f57832a9778364d69206aa53e221fc2eb19250f2f57809b3c8"} Nov 01 22:22:05.869459 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:05.868913 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerStarted","Data":"5742f5547606f8faf8447723d3453e283e4b427d14d4f1564a7ec6673df57e1d"} Nov 01 22:22:05.884629 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:05.884549 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9pqxl" podStartSLOduration=2.256451124 podStartE2EDuration="10.884534264s" totalImagesPullingTime="8.62808314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:04.889620042 +0000 UTC m=+10.791655882" watchObservedRunningTime="2026-11-01 22:22:05.884534264 +0000 UTC m=+11.786570106" Nov 01 22:22:05.901722 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:05.901647 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podStartSLOduration=2.009164937 podStartE2EDuration="10.9016304s" totalImagesPullingTime="8.892465463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:05.901613173 +0000 UTC m=+11.803649014" watchObservedRunningTime="2026-11-01 22:22:05.9016304 +0000 UTC m=+11.803666237" Nov 01 22:22:06.650596 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:06.650566 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:06.650717 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:06.650695 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:06.873130 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:06.873084 2576 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="880795c0e9353ca6d7076cd6da0b9a2037f6ac83e1bcd5b6a082bae4e21df18f" exitCode=0 Nov 01 22:22:06.873308 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:06.873175 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86mt" event={"ID":"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0","Type":"ContainerDied","Data":"880795c0e9353ca6d7076cd6da0b9a2037f6ac83e1bcd5b6a082bae4e21df18f"} Nov 01 22:22:06.874882 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:06.874859 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zlskb" event={"ID":"665f24e2-f8fb-4bc7-8958-191e776f78fd","Type":"ContainerStarted","Data":"2fb90cb250ba96a50f83481969b606fec356f4fed93a38b233ac0920b10e71be"} Nov 01 22:22:06.876711 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:06.876685 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pchz7" event={"ID":"cedefae3-8d26-4f04-a6c1-78d62920c2d6","Type":"ContainerStarted","Data":"7726c83977ca9d9e2f4169379903d476b7ac64549f58f6e1218959a0f95e9451"} Nov 01 22:22:06.906845 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:06.906751 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pchz7" podStartSLOduration=3.335058604 podStartE2EDuration="11.906736772s" totalImagesPullingTime="8.571678168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:06.90616162 +0000 UTC m=+12.808197460" watchObservedRunningTime="2026-11-01 22:22:06.906736772 +0000 UTC m=+12.808772612" Nov 01 22:22:06.921182 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:06.921134 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zlskb" podStartSLOduration=1.487514751 podStartE2EDuration="11.921102332s" totalImagesPullingTime="10.433587581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:06.920602889 +0000 UTC m=+12.822638730" watchObservedRunningTime="2026-11-01 22:22:06.921102332 +0000 UTC m=+12.823138174" Nov 01 22:22:07.650463 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:07.650428 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:07.650651 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:07.650476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:07.650651 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:07.650546 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:07.650737 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:07.650683 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:08.651059 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:08.651026 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:08.651849 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:08.651152 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:08.883968 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:08.883934 2576 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="5576473d3b4ceae76515a6a27b3af71cfbf821f941eaa15992a66f148d3cbb8c" exitCode=0 Nov 01 22:22:08.884140 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:08.883982 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86mt" event={"ID":"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0","Type":"ContainerDied","Data":"5576473d3b4ceae76515a6a27b3af71cfbf821f941eaa15992a66f148d3cbb8c"} Nov 01 22:22:09.651185 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:09.651151 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:09.651669 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:09.651228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:09.651669 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:09.651352 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:09.651669 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:09.651498 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:10.650362 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:10.650324 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:10.650525 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:10.650467 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:11.262954 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.262918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:11.263370 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.262978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:11.263370 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.263091 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:11.263370 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.263094 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:11.263370 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.263138 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:11.263370 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.263153 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gwb7v for pod openshift-network-diagnostics/network-check-target-h5fn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:11.263370 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.263156 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret podName:330ca60e-5fa1-4a41-8617-73450d13cac2 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:27.263140809 +0000 UTC m=+33.165176641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret") pod "global-pull-secret-syncer-cgcvk" (UID: "330ca60e-5fa1-4a41-8617-73450d13cac2") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:11.263370 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.263218 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v podName:5029000f-c7ae-4f6f-9b0c-5240e385f697 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:27.263205431 +0000 UTC m=+33.165241250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwb7v" (UniqueName: "kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v") pod "network-check-target-h5fn2" (UID: "5029000f-c7ae-4f6f-9b0c-5240e385f697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:11.363971 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.363932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:11.364183 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.364097 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:11.364251 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.364194 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs podName:7c8ec790-a230-403d-8972-5634c7d2ae0f nodeName:}" failed. No retries permitted until 2026-11-01 22:22:27.364177399 +0000 UTC m=+33.266213234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs") pod "network-metrics-daemon-mjw8q" (UID: "7c8ec790-a230-403d-8972-5634c7d2ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:11.650350 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.650323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:11.650461 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.650433 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:11.650546 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.650484 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:11.650596 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:11.650575 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:11.893865 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.893687 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wbfqp" event={"ID":"cdc57449-d2a8-41fa-808b-6ef9a3517bc6","Type":"ContainerStarted","Data":"da89570e3078a40b279bd624fb8c338f7c1692832c3794c15747c3fc5915775f"} Nov 01 22:22:11.896562 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.896530 2576 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="2eda56738be3c0efa10173b8b504b3e5f65317e451c0192f3fb12c27ea25e52a" exitCode=0 Nov 01 22:22:11.896681 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.896617 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86mt" event={"ID":"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0","Type":"ContainerDied","Data":"2eda56738be3c0efa10173b8b504b3e5f65317e451c0192f3fb12c27ea25e52a"} Nov 01 22:22:11.900298 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.900273 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" event={"ID":"d35d0387-f298-43a2-977a-51386c17077a","Type":"ContainerStarted","Data":"b643ce813e34e4662c4276fe1051423734a9adf24fe88070b5a2816e1c221c94"} Nov 01 22:22:11.900376 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.900349 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" event={"ID":"d35d0387-f298-43a2-977a-51386c17077a","Type":"ContainerStarted","Data":"0f809f547be9b42e753bd35abd89c7a45cbbd6be602605e50c6728e93496018f"} Nov 01 22:22:11.900409 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.900399 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" event={"ID":"d35d0387-f298-43a2-977a-51386c17077a","Type":"ContainerStarted","Data":"d7ec9d009b532f11871caf6e33bcc726e2db42ab0718a42c4032544c41227ca1"} Nov 01 22:22:11.900450 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.900443 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" event={"ID":"d35d0387-f298-43a2-977a-51386c17077a","Type":"ContainerStarted","Data":"1ac86afd60f5df661583e6e5c9c6d489ff0b3d4f2c3d2e8b6bc5ba7932a5d48a"} Nov 01 22:22:11.900499 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.900487 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" event={"ID":"d35d0387-f298-43a2-977a-51386c17077a","Type":"ContainerStarted","Data":"27e57c0e1cb081dffc3302be8743ce2364b44923aea43ac08f74f6d5337d2e03"} Nov 01 22:22:11.900543 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.900532 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" event={"ID":"d35d0387-f298-43a2-977a-51386c17077a","Type":"ContainerStarted","Data":"eeed599882393ad7067196c059068d2fc4823a48a455d5ee2c94986d762747e8"} Nov 01 22:22:11.909146 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:11.909091 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wbfqp" podStartSLOduration=1.113908025 podStartE2EDuration="16.909080037s" totalImagesPullingTime="15.795172012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-01 22:22:11.908897803 +0000 UTC m=+17.810933644" watchObservedRunningTime="2026-11-01 22:22:11.909080037 +0000 UTC m=+17.811115875" Nov 01 22:22:12.650857 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:12.650826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:12.651311 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:12.650954 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:12.905176 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:12.905091 2576 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="7521b61ef36504f999e4c8ca3be403b42782024c87da483463b076d85fc09737" exitCode=0 Nov 01 22:22:12.905176 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:12.905146 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86mt" event={"ID":"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0","Type":"ContainerDied","Data":"7521b61ef36504f999e4c8ca3be403b42782024c87da483463b076d85fc09737"} Nov 01 22:22:13.650303 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:13.650275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:13.650440 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:13.650373 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:13.650440 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:13.650399 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:13.650525 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:13.650445 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:13.911817 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:13.911764 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s86mt" event={"ID":"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0","Type":"ContainerStarted","Data":"18ee284612b310b7c961cdc60a948d9aaed4a255f0ccd201c221a7120cf6fd75"} Nov 01 22:22:13.915158 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:13.915135 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" event={"ID":"d35d0387-f298-43a2-977a-51386c17077a","Type":"ContainerStarted","Data":"9f195539b13f662eec1984f2eb0cf5c8934582ce7c3569bb2b6518c1e1a1b53f"} Nov 01 22:22:13.933067 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:13.933023 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s86mt" podStartSLOduration=7.798452152 podStartE2EDuration="18.933011383s" totalImagesPullingTime="11.134559231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:13.932461296 +0000 UTC m=+19.834497137" watchObservedRunningTime="2026-11-01 22:22:13.933011383 +0000 UTC m=+19.835047223" Nov 01 22:22:14.651627 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:14.651598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:14.651761 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:14.651678 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:15.650610 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:15.650573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:15.651141 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:15.650579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:15.651141 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:15.650671 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:15.651141 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:15.650751 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:16.650504 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:16.650337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:16.650739 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:16.650589 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:16.923948 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:16.923865 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" event={"ID":"d35d0387-f298-43a2-977a-51386c17077a","Type":"ContainerStarted","Data":"db38fb4bae4b93e304e91322fcfa33a64e20980e5b28bd2cb80d644f89367c14"} Nov 01 22:22:16.924158 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:16.924134 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:22:16.924239 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:16.924203 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:22:16.939800 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:16.939776 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:22:16.939929 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:16.939878 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:22:16.949464 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:16.949424 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" podStartSLOduration=6.362241064 podStartE2EDuration="21.949414337s" totalImagesPullingTime="15.587173273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:16.948758147 +0000 UTC m=+22.850793988" watchObservedRunningTime="2026-11-01 22:22:16.949414337 +0000 UTC m=+22.851450176" Nov 01 22:22:17.650444 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:17.650414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:17.650628 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:17.650453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:17.650628 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:17.650569 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:17.650742 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:17.650692 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:17.651264 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:17.651240 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mjw8q"] Nov 01 22:22:17.654492 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:17.654425 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cgcvk"] Nov 01 22:22:17.654600 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:17.654522 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:17.654684 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:17.654616 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:17.655011 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:17.654992 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h5fn2"] Nov 01 22:22:17.926249 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:17.926173 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:17.926249 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:17.926188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:17.926423 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:17.926253 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:17.926423 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:17.926290 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:17.926726 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:17.926704 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:18.928190 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:18.928163 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:19.172618 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:19.172593 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:22:19.650417 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:19.650385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:19.650417 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:19.650411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:19.650623 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:19.650485 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:19.650623 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:19.650508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:19.650623 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:19.650614 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:19.651217 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:19.650689 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:21.650357 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:21.650324 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:21.650357 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:21.650343 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:21.650750 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:21.650325 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:21.650750 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:21.650421 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:21.650750 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:21.650494 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:21.650750 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:21.650552 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:23.650288 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:23.650251 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:23.650655 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:23.650346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:23.650655 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:23.650352 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:23.650655 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:23.650369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:23.650655 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:23.650432 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:23.650655 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:23.650480 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:24.057796 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:24.057772 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9pqxl_5265e1c6-2a08-4439-b6a9-42ed2fdbd81a/dns-node-resolver/0.log" Nov 01 22:22:24.482300 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:24.482221 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:22:24.482450 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:24.482384 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:24.482805 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:24.482785 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:22:25.428292 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:25.428265 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wbfqp_cdc57449-d2a8-41fa-808b-6ef9a3517bc6/node-ca/0.log" Nov 01 22:22:25.650276 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:25.650246 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:25.650416 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:25.650317 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:25.650416 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:25.650338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:25.650485 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:25.650412 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:25.650516 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:25.650475 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:25.650552 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:25.650532 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:27.289728 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:27.289694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:27.290108 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:27.289739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:27.290108 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.289824 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:27.290108 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.289854 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:27.290108 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.289874 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret podName:330ca60e-5fa1-4a41-8617-73450d13cac2 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.289860341 +0000 UTC m=+65.191896159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret") pod "global-pull-secret-syncer-cgcvk" (UID: "330ca60e-5fa1-4a41-8617-73450d13cac2") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:27.290108 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.289873 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:27.290108 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.289889 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gwb7v for pod openshift-network-diagnostics/network-check-target-h5fn2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:27.290108 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.289912 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v podName:5029000f-c7ae-4f6f-9b0c-5240e385f697 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.289905719 +0000 UTC m=+65.191941537 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gwb7v" (UniqueName: "kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v") pod "network-check-target-h5fn2" (UID: "5029000f-c7ae-4f6f-9b0c-5240e385f697") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:27.390475 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:27.390446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:27.390607 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.390544 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:27.390607 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.390588 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs podName:7c8ec790-a230-403d-8972-5634c7d2ae0f nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.390575874 +0000 UTC m=+65.292611691 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs") pod "network-metrics-daemon-mjw8q" (UID: "7c8ec790-a230-403d-8972-5634c7d2ae0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:27.650986 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:27.650907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:27.650986 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:27.650945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:27.651225 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:27.650907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:27.651225 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.651019 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:27.651225 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.651084 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:27.651225 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:27.651172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:29.650963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:29.650931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:29.651349 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:29.650977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:29.651349 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:29.650985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:29.651349 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:29.651073 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:29.651349 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:29.651194 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:29.651349 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:29.651282 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:31.650165 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:31.650135 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:31.650165 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:31.650167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:31.650589 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:31.650242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:31.650589 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:31.650246 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:31.650589 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:31.650321 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:31.650589 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:31.650397 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:33.650954 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:33.650917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:33.650954 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:33.650944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:33.650954 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:33.650955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:33.651600 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:33.651056 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:33.651600 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:33.651178 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:33.651600 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:33.651239 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:35.650130 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:35.650091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:35.650602 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:35.650137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:35.650602 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:35.650209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:35.650602 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:35.650214 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:35.650602 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:35.650315 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:35.650602 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:35.650415 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:37.650281 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:37.650249 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:37.650281 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:37.650275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:37.650681 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:37.650337 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" Nov 01 22:22:37.650681 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:37.650373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:37.650681 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:37.650450 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjw8q" podUID="7c8ec790-a230-403d-8972-5634c7d2ae0f" Nov 01 22:22:37.650681 ip-10-0-128-55 kubenswrapper[2576]: E1101 22:22:37.650506 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cgcvk" podUID="330ca60e-5fa1-4a41-8617-73450d13cac2" Nov 01 22:22:38.949811 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:38.949786 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeReady" Nov 01 22:22:38.950189 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:38.949906 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 22:22:38.995015 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:38.994993 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9xvw2"] Nov 01 22:22:38.996699 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:38.996686 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:22:38.997985 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:38.997962 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b6vcp"] Nov 01 22:22:39.000257 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.000239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.004328 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.004296 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.004328 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.004313 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.004561 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.004520 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4jwn7\"" Nov 01 22:22:39.004561 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.004552 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:22:39.008455 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.008437 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4bt99\"" Nov 01 22:22:39.008683 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.008666 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:22:39.009265 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.009245 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9xvw2"] Nov 01 22:22:39.009926 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.009865 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.009926 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.009874 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.010068 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.009925 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:22:39.015705 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.013245 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b6vcp"] Nov 01 22:22:39.015705 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.015465 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-78mfv"] Nov 01 22:22:39.017950 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.017931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.030828 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.030789 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78mfv"] Nov 01 22:22:39.073664 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.073612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/800f0c00-7f68-419a-96c5-6b947a2101a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.073664 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.073647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.073775 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.073730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szx57\" (UniqueName: \"kubernetes.io/projected/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-api-access-szx57\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.073775 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.073760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/800f0c00-7f68-419a-96c5-6b947a2101a5-crio-socket\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.073839 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.073775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/800f0c00-7f68-419a-96c5-6b947a2101a5-data-volume\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.073839 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.073804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1a395fe-2467-4736-9fd3-bab535023c24-cert\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:22:39.073914 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.073902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4n8\" (UniqueName: \"kubernetes.io/projected/f1a395fe-2467-4736-9fd3-bab535023c24-kube-api-access-rq4n8\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:22:39.105073 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.105045 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:22:39.105225 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.105209 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:22:39.105300 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.105267 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4kglz\"" Nov 01 22:22:39.174831 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.174801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljl7r\" (UniqueName: \"kubernetes.io/projected/8c95cb60-d629-48ef-859d-85d7893afe81-kube-api-access-ljl7r\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.174932 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.174840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1a395fe-2467-4736-9fd3-bab535023c24-cert\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:22:39.174932 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.174863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4n8\" (UniqueName: \"kubernetes.io/projected/f1a395fe-2467-4736-9fd3-bab535023c24-kube-api-access-rq4n8\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:22:39.175037 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.174940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c95cb60-d629-48ef-859d-85d7893afe81-config-volume\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.175037 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.174971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95cb60-d629-48ef-859d-85d7893afe81-tmp-dir\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.175161 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.175045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/800f0c00-7f68-419a-96c5-6b947a2101a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.175161 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.175089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.175242 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.175157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c95cb60-d629-48ef-859d-85d7893afe81-metrics-tls\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.175242 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.175193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szx57\" (UniqueName: \"kubernetes.io/projected/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-api-access-szx57\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.175337 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.175246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/800f0c00-7f68-419a-96c5-6b947a2101a5-crio-socket\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.175337 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.175271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/800f0c00-7f68-419a-96c5-6b947a2101a5-data-volume\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.175436 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.175396 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/800f0c00-7f68-419a-96c5-6b947a2101a5-crio-socket\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.175571 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.175554 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/800f0c00-7f68-419a-96c5-6b947a2101a5-data-volume\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.175691 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.175675 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.178291 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.178269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/800f0c00-7f68-419a-96c5-6b947a2101a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.178364 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.178274 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1a395fe-2467-4736-9fd3-bab535023c24-cert\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:22:39.182615 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.182590 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szx57\" (UniqueName: \"kubernetes.io/projected/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-api-access-szx57\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.182850 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.182834 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4n8\" (UniqueName: \"kubernetes.io/projected/f1a395fe-2467-4736-9fd3-bab535023c24-kube-api-access-rq4n8\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:22:39.275928 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.275903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c95cb60-d629-48ef-859d-85d7893afe81-metrics-tls\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.276027 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.275937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljl7r\" (UniqueName: \"kubernetes.io/projected/8c95cb60-d629-48ef-859d-85d7893afe81-kube-api-access-ljl7r\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.276027 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.275964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c95cb60-d629-48ef-859d-85d7893afe81-config-volume\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.276027 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.275978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95cb60-d629-48ef-859d-85d7893afe81-tmp-dir\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.276340 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.276320 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95cb60-d629-48ef-859d-85d7893afe81-tmp-dir\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.276523 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.276502 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c95cb60-d629-48ef-859d-85d7893afe81-config-volume\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.277963 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.277943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c95cb60-d629-48ef-859d-85d7893afe81-metrics-tls\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.283452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.283429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljl7r\" (UniqueName: \"kubernetes.io/projected/8c95cb60-d629-48ef-859d-85d7893afe81-kube-api-access-ljl7r\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.309001 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.308978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:22:39.321308 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.321285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:22:39.328023 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.328002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:39.586548 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.586521 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9xvw2"] Nov 01 22:22:39.591696 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:22:39.591660 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a395fe_2467_4736_9fd3_bab535023c24.slice/crio-e3a1dd4458820e8d696fedc27c0f89ef90ed11170e67062fe7dda115aa57cdf7 WatchSource:0}: Error finding container e3a1dd4458820e8d696fedc27c0f89ef90ed11170e67062fe7dda115aa57cdf7: Status 404 returned error can't find the container with id e3a1dd4458820e8d696fedc27c0f89ef90ed11170e67062fe7dda115aa57cdf7 Nov 01 22:22:39.597634 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.597604 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b6vcp"] Nov 01 22:22:39.600000 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:22:39.599978 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800f0c00_7f68_419a_96c5_6b947a2101a5.slice/crio-2670bb4c5609c3a48a9734677e5a4765e5bb54bbd08121afd1ba70013b636a81 WatchSource:0}: Error finding container 2670bb4c5609c3a48a9734677e5a4765e5bb54bbd08121afd1ba70013b636a81: Status 404 returned error can't find the container with id 2670bb4c5609c3a48a9734677e5a4765e5bb54bbd08121afd1ba70013b636a81 Nov 01 22:22:39.613006 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.612982 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78mfv"] Nov 01 22:22:39.634081 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:22:39.634060 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c95cb60_d629_48ef_859d_85d7893afe81.slice/crio-6acca285087cd2fcab46e0e1307b6b05ce41f106c393e4abb68cba8c8c2e4c46 WatchSource:0}: Error finding container 6acca285087cd2fcab46e0e1307b6b05ce41f106c393e4abb68cba8c8c2e4c46: Status 404 returned error can't find the container with id 6acca285087cd2fcab46e0e1307b6b05ce41f106c393e4abb68cba8c8c2e4c46 Nov 01 22:22:39.650951 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.650931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:39.651058 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.651024 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:39.651058 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.651036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:39.654570 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.654552 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:22:39.654666 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.654647 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:22:39.655607 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.655595 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9pl6q\"" Nov 01 22:22:39.655740 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.655724 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:22:39.655952 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.655942 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fpxpg\"" Nov 01 22:22:39.656065 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.656050 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5scqq\"" Nov 01 22:22:39.656106 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.656053 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:22:39.742194 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.742175 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9pqxl_5265e1c6-2a08-4439-b6a9-42ed2fdbd81a/dns-node-resolver/0.log" Nov 01 22:22:39.971242 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.971158 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78mfv" event={"ID":"8c95cb60-d629-48ef-859d-85d7893afe81","Type":"ContainerStarted","Data":"6acca285087cd2fcab46e0e1307b6b05ce41f106c393e4abb68cba8c8c2e4c46"} Nov 01 22:22:39.972549 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.972526 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b6vcp" event={"ID":"800f0c00-7f68-419a-96c5-6b947a2101a5","Type":"ContainerStarted","Data":"c7806935e2c71c888ac2baa2c50c6edc0c8480047776460060d63f56e2d3f9e8"} Nov 01 22:22:39.972651 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.972578 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b6vcp" event={"ID":"800f0c00-7f68-419a-96c5-6b947a2101a5","Type":"ContainerStarted","Data":"2670bb4c5609c3a48a9734677e5a4765e5bb54bbd08121afd1ba70013b636a81"} Nov 01 22:22:39.973620 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:39.973600 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9xvw2" event={"ID":"f1a395fe-2467-4736-9fd3-bab535023c24","Type":"ContainerStarted","Data":"e3a1dd4458820e8d696fedc27c0f89ef90ed11170e67062fe7dda115aa57cdf7"} Nov 01 22:22:40.016492 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.016465 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56fb5c6985-4cddf"] Nov 01 22:22:40.018016 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.018002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.024148 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.024131 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 22:22:40.024248 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.024131 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 22:22:40.024248 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.024222 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-w2wlc\"" Nov 01 22:22:40.024498 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.024480 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 22:22:40.024555 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.024493 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 22:22:40.024555 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.024506 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 22:22:40.024641 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.024575 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 22:22:40.024831 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.024819 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 22:22:40.029715 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.029695 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fb5c6985-4cddf"] Nov 01 22:22:40.182780 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.182751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprfw\" (UniqueName: \"kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.182952 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.182825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.182952 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.182855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.182952 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.182884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.182952 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.182909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.183187 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.182956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.183187 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.182984 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.283686 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.283646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.283847 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.283694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.283847 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.283720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.283847 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.283772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.283847 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.283804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.284029 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.283846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprfw\" (UniqueName: \"kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.284029 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.283911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.285047 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.285023 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.285180 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.285063 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.285180 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.285150 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.285382 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.285363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.287825 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.287799 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.289389 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.289368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.294280 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.294255 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprfw\" (UniqueName: \"kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.328014 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.327988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:40.613360 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.613286 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fb5c6985-4cddf"] Nov 01 22:22:40.616938 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:22:40.616906 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5e5509_37bc_4415_9a55_3b85cb87511f.slice/crio-f391300cdeddeff794386ff4c2bbf9a9be6da8f6aceb6daf5cddb1c4d7a216c0 WatchSource:0}: Error finding container f391300cdeddeff794386ff4c2bbf9a9be6da8f6aceb6daf5cddb1c4d7a216c0: Status 404 returned error can't find the container with id f391300cdeddeff794386ff4c2bbf9a9be6da8f6aceb6daf5cddb1c4d7a216c0 Nov 01 22:22:40.978009 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.977922 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fb5c6985-4cddf" event={"ID":"ef5e5509-37bc-4415-9a55-3b85cb87511f","Type":"ContainerStarted","Data":"f391300cdeddeff794386ff4c2bbf9a9be6da8f6aceb6daf5cddb1c4d7a216c0"} Nov 01 22:22:40.979918 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:40.979890 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b6vcp" event={"ID":"800f0c00-7f68-419a-96c5-6b947a2101a5","Type":"ContainerStarted","Data":"28bb44f5c269cfc434edf4335cd61e631d90650f3229e3d71ab3fbee6648081e"} Nov 01 22:22:41.343005 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.342979 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wbfqp_cdc57449-d2a8-41fa-808b-6ef9a3517bc6/node-ca/0.log" Nov 01 22:22:41.577290 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.577214 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc"] Nov 01 22:22:41.579188 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.579165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.583736 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.583711 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 22:22:41.587914 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.587894 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 22:22:41.587988 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.587965 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 22:22:41.588048 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.588037 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 22:22:41.588391 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.588232 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 22:22:41.588391 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.588366 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h46mq\"" Nov 01 22:22:41.590403 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.590345 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc"] Nov 01 22:22:41.697589 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.697554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.697750 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.697643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b137d23-cef1-4c5b-975d-3902be4f9f83-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.697750 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.697706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425j7\" (UniqueName: \"kubernetes.io/projected/3b137d23-cef1-4c5b-975d-3902be4f9f83-kube-api-access-425j7\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.697832 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.697753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.798052 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.798017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425j7\" (UniqueName: \"kubernetes.io/projected/3b137d23-cef1-4c5b-975d-3902be4f9f83-kube-api-access-425j7\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.798222 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.798086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.798222 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.798133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.798222 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.798207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b137d23-cef1-4c5b-975d-3902be4f9f83-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.798934 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.798903 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b137d23-cef1-4c5b-975d-3902be4f9f83-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.801314 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.801291 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.801712 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.801692 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.807097 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.807075 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-425j7\" (UniqueName: \"kubernetes.io/projected/3b137d23-cef1-4c5b-975d-3902be4f9f83-kube-api-access-425j7\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.895864 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.895778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:22:41.988414 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.988385 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78mfv" event={"ID":"8c95cb60-d629-48ef-859d-85d7893afe81","Type":"ContainerStarted","Data":"7cc093428401ea8686e757cc077f9c20d56b4dc666f82fd919805e4f6b27b82a"} Nov 01 22:22:41.988819 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.988447 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78mfv" event={"ID":"8c95cb60-d629-48ef-859d-85d7893afe81","Type":"ContainerStarted","Data":"0a0377efbe67ee85fb26a7323434decd828888819fc671a1a5e369b688a8b81a"} Nov 01 22:22:41.989241 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:41.989220 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:42.177783 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.177680 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-78mfv" podStartSLOduration=2.536243989 podStartE2EDuration="4.177660297s" totalImagesPullingTime="1.641416308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:42.007261575 +0000 UTC m=+47.909297413" watchObservedRunningTime="2026-11-01 22:22:42.177660297 +0000 UTC m=+48.079696134" Nov 01 22:22:42.177875 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.177859 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc"] Nov 01 22:22:42.180364 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:22:42.180331 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b137d23_cef1_4c5b_975d_3902be4f9f83.slice/crio-22905f1fe76012df0956db1160fe07d3432f44a9e43b6b7d3250d9a056804d87 WatchSource:0}: Error finding container 22905f1fe76012df0956db1160fe07d3432f44a9e43b6b7d3250d9a056804d87: Status 404 returned error can't find the container with id 22905f1fe76012df0956db1160fe07d3432f44a9e43b6b7d3250d9a056804d87 Nov 01 22:22:42.245791 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.245764 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59557fbb88-l56pg"] Nov 01 22:22:42.248211 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.248187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.256788 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.256066 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 22:22:42.259371 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.259313 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59557fbb88-l56pg"] Nov 01 22:22:42.403521 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.403484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.403690 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.403547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.403690 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.403638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.403794 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.403695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.403794 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.403744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.403919 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.403900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.403956 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.403943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvzs\" (UniqueName: \"kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.404006 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.403981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.504915 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.504882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.504932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvzs\" (UniqueName: \"kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.504970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.505009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.505051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505096 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.505076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505939 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.505105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505939 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.505154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505939 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.505640 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505939 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.505859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.505939 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.505922 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.506153 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.506131 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.506420 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.506397 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.508381 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.508344 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.508482 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.508463 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.512943 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.512921 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvzs\" (UniqueName: \"kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.561555 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.561523 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:42.836643 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.836568 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59557fbb88-l56pg"] Nov 01 22:22:42.840986 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:22:42.840956 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6f0fd2_0f11_4cc4_abe2_169f81abd94b.slice/crio-e2a0c7451a9c49a26911dcf57cbae6f93fc5c63c885ae476cffdaa0b3df3fd0b WatchSource:0}: Error finding container e2a0c7451a9c49a26911dcf57cbae6f93fc5c63c885ae476cffdaa0b3df3fd0b: Status 404 returned error can't find the container with id e2a0c7451a9c49a26911dcf57cbae6f93fc5c63c885ae476cffdaa0b3df3fd0b Nov 01 22:22:42.993756 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.993707 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b6vcp" event={"ID":"800f0c00-7f68-419a-96c5-6b947a2101a5","Type":"ContainerStarted","Data":"6c584086cfbf8afe3dcdf87bad62e7636b040fa5ed41e4ef0e5db70d0548a23e"} Nov 01 22:22:42.995263 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.995234 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59557fbb88-l56pg" event={"ID":"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b","Type":"ContainerStarted","Data":"e2a0c7451a9c49a26911dcf57cbae6f93fc5c63c885ae476cffdaa0b3df3fd0b"} Nov 01 22:22:42.996844 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:42.996816 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" event={"ID":"3b137d23-cef1-4c5b-975d-3902be4f9f83","Type":"ContainerStarted","Data":"22905f1fe76012df0956db1160fe07d3432f44a9e43b6b7d3250d9a056804d87"} Nov 01 22:22:44.668332 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:44.668279 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b6vcp" podStartSLOduration=4.13999679 podStartE2EDuration="6.668265106s" totalImagesPullingTime="2.528268316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:43.011599647 +0000 UTC m=+48.913635489" watchObservedRunningTime="2026-11-01 22:22:44.668265106 +0000 UTC m=+50.570300946" Nov 01 22:22:45.006349 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:45.006301 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fb5c6985-4cddf" event={"ID":"ef5e5509-37bc-4415-9a55-3b85cb87511f","Type":"ContainerStarted","Data":"e8cf9dda765cb973156e028af260a3afb507ee05e012fe02afa01ccb5083d3e4"} Nov 01 22:22:45.008562 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:45.008532 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" event={"ID":"3b137d23-cef1-4c5b-975d-3902be4f9f83","Type":"ContainerStarted","Data":"a380432309a97ee3c8ff994a676ba2a0ae58b9019cc9065ff9e61491f56e124c"} Nov 01 22:22:45.008689 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:45.008588 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" event={"ID":"3b137d23-cef1-4c5b-975d-3902be4f9f83","Type":"ContainerStarted","Data":"63cc88087d2527a3b70c87ccf75b8a7724b2448efa5027869ca902f2319eb458"} Nov 01 22:22:45.024035 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:45.023977 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56fb5c6985-4cddf" podStartSLOduration=2.669049712 podStartE2EDuration="6.023961624s" totalImagesPullingTime="3.354911912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:45.023235087 +0000 UTC m=+50.925270928" watchObservedRunningTime="2026-11-01 22:22:45.023961624 +0000 UTC m=+50.925997464" Nov 01 22:22:46.012669 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.012576 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9xvw2" event={"ID":"f1a395fe-2467-4736-9fd3-bab535023c24","Type":"ContainerStarted","Data":"23b67d392abbae04a136b72335fb690c5c85e8f3efe5573a52be31cb5c028616"} Nov 01 22:22:46.014047 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.014012 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59557fbb88-l56pg" event={"ID":"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b","Type":"ContainerStarted","Data":"4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe"} Nov 01 22:22:46.030211 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.030167 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9xvw2" podStartSLOduration=2.46766811 podStartE2EDuration="8.030155139s" totalImagesPullingTime="5.562487029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:46.028935032 +0000 UTC m=+51.930970873" watchObservedRunningTime="2026-11-01 22:22:46.030155139 +0000 UTC m=+51.932190978" Nov 01 22:22:46.030486 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.030465 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" podStartSLOduration=3.247577277 podStartE2EDuration="5.030459782s" totalImagesPullingTime="1.782882505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:45.040048684 +0000 UTC m=+50.942084525" watchObservedRunningTime="2026-11-01 22:22:46.030459782 +0000 UTC m=+51.932495622" Nov 01 22:22:46.045071 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.045022 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59557fbb88-l56pg" podStartSLOduration=1.1442853450000001 podStartE2EDuration="4.045004666s" totalImagesPullingTime="2.900719321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:46.044862945 +0000 UTC m=+51.946898798" watchObservedRunningTime="2026-11-01 22:22:46.045004666 +0000 UTC m=+51.947040507" Nov 01 22:22:46.955580 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.955543 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8flmn"] Nov 01 22:22:46.957874 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.957848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:46.967724 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.967697 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 22:22:46.967724 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.967718 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zj5nt\"" Nov 01 22:22:46.968089 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.968072 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 22:22:46.968161 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:46.968088 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 22:22:47.045452 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.045427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-wtmp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.045857 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.045488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-textfile\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.045857 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.045565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-metrics-client-ca\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.045857 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.045628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.045857 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.045656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-tls\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.045857 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.045725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-accelerators-collector-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.045857 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.045769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-root\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.045857 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.045819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdqp\" (UniqueName: \"kubernetes.io/projected/7d398858-b6ca-494f-a576-4456c4287992-kube-api-access-7mdqp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.046137 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.045910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-sys\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.146683 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.146656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-wtmp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.146815 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.146700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-textfile\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.146867 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.146825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-metrics-client-ca\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.146926 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.146826 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-wtmp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147043 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147027 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-textfile\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147149 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-tls\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147253 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-accelerators-collector-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147304 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-root\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147356 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdqp\" (UniqueName: \"kubernetes.io/projected/7d398858-b6ca-494f-a576-4456c4287992-kube-api-access-7mdqp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147406 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-sys\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147406 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147376 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-root\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147498 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-sys\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147498 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147477 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-metrics-client-ca\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.147692 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.147671 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-accelerators-collector-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.149647 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.149625 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-tls\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.149733 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.149658 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.156952 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.156923 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdqp\" (UniqueName: \"kubernetes.io/projected/7d398858-b6ca-494f-a576-4456c4287992-kube-api-access-7mdqp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.271427 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:47.271396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:22:47.282522 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:22:47.282492 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d398858_b6ca_494f_a576_4456c4287992.slice/crio-bac89d42d44a8e9d1601d1a902dc297ac4930777254e0fd2f4ba473f2a852a57 WatchSource:0}: Error finding container bac89d42d44a8e9d1601d1a902dc297ac4930777254e0fd2f4ba473f2a852a57: Status 404 returned error can't find the container with id bac89d42d44a8e9d1601d1a902dc297ac4930777254e0fd2f4ba473f2a852a57 Nov 01 22:22:48.025454 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:48.025410 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8flmn" event={"ID":"7d398858-b6ca-494f-a576-4456c4287992","Type":"ContainerStarted","Data":"bac89d42d44a8e9d1601d1a902dc297ac4930777254e0fd2f4ba473f2a852a57"} Nov 01 22:22:49.029864 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:49.029830 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d398858-b6ca-494f-a576-4456c4287992" containerID="562b6829d92dd297a61f6640f7fe4b808f1071e1c4421450b6458a6b025dfee5" exitCode=0 Nov 01 22:22:49.030302 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:49.029896 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8flmn" event={"ID":"7d398858-b6ca-494f-a576-4456c4287992","Type":"ContainerDied","Data":"562b6829d92dd297a61f6640f7fe4b808f1071e1c4421450b6458a6b025dfee5"} Nov 01 22:22:49.941140 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:49.941100 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:22:50.038206 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:50.038174 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8flmn" event={"ID":"7d398858-b6ca-494f-a576-4456c4287992","Type":"ContainerStarted","Data":"c0ec7e7386e005d73f5a5cfa0d8eed5af6f69b56eddaa5974d878a5817599748"} Nov 01 22:22:50.038594 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:50.038228 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8flmn" event={"ID":"7d398858-b6ca-494f-a576-4456c4287992","Type":"ContainerStarted","Data":"ff5ba22419e01726c62089d76ec7cc9db9ee05ac664828010820cbef0c7b7df4"} Nov 01 22:22:50.058859 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:50.058815 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8flmn" podStartSLOduration=2.713350881 podStartE2EDuration="4.058801799s" totalImagesPullingTime="1.28097005s" totalInitContainerRuntime="64.480868ms" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:50.058030418 +0000 UTC m=+55.960066257" watchObservedRunningTime="2026-11-01 22:22:50.058801799 +0000 UTC m=+55.960837640" Nov 01 22:22:50.328616 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:50.328580 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:50.328809 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:50.328650 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:50.333405 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:50.333383 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:51.044856 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.044828 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:22:51.747573 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.747534 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8"] Nov 01 22:22:51.749698 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.749677 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:51.757403 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.757381 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mjnqw\"" Nov 01 22:22:51.757498 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.757459 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 22:22:51.760953 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.760931 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8"] Nov 01 22:22:51.885900 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.885861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7059b374-3864-4296-bd7b-d77161dfca28-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:51.886069 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.885967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfb9k\" (UniqueName: \"kubernetes.io/projected/7059b374-3864-4296-bd7b-d77161dfca28-kube-api-access-cfb9k\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:51.987257 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.987221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfb9k\" (UniqueName: \"kubernetes.io/projected/7059b374-3864-4296-bd7b-d77161dfca28-kube-api-access-cfb9k\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:51.987397 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.987290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7059b374-3864-4296-bd7b-d77161dfca28-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:51.989744 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.989712 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7059b374-3864-4296-bd7b-d77161dfca28-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:51.995360 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:51.995339 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfb9k\" (UniqueName: \"kubernetes.io/projected/7059b374-3864-4296-bd7b-d77161dfca28-kube-api-access-cfb9k\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:52.060041 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:52.059955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:52.326371 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:52.326349 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8"] Nov 01 22:22:52.328725 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:22:52.328692 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7059b374_3864_4296_bd7b_d77161dfca28.slice/crio-bf754872a4c205f57cadafe91097ed7afb1912310227beb2a3a881cdf820968e WatchSource:0}: Error finding container bf754872a4c205f57cadafe91097ed7afb1912310227beb2a3a881cdf820968e: Status 404 returned error can't find the container with id bf754872a4c205f57cadafe91097ed7afb1912310227beb2a3a881cdf820968e Nov 01 22:22:52.562189 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:52.562160 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:52.562382 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:52.562317 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:52.567135 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:52.567099 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:53.001991 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.001962 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-78mfv" Nov 01 22:22:53.049860 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.049818 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" event={"ID":"7059b374-3864-4296-bd7b-d77161dfca28","Type":"ContainerStarted","Data":"bf754872a4c205f57cadafe91097ed7afb1912310227beb2a3a881cdf820968e"} Nov 01 22:22:53.054915 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.054872 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:22:53.101739 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.101702 2576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56fb5c6985-4cddf"] Nov 01 22:22:53.207269 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.205169 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:22:53.208248 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.208228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.212450 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.212412 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 22:22:53.212775 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.212634 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 22:22:53.212775 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.212650 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2acjdgdf71rol\"" Nov 01 22:22:53.212775 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.212681 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4fw2s\"" Nov 01 22:22:53.212775 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.212697 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 22:22:53.217842 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.216733 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 22:22:53.217842 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.216750 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 22:22:53.217842 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.216822 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 22:22:53.218299 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.218262 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 22:22:53.220417 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.218922 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 22:22:53.220417 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.219474 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 22:22:53.220558 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.220449 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 22:22:53.221061 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.220648 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 22:22:53.221163 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.221095 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 22:22:53.221163 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.221105 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 22:22:53.221163 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.221130 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 22:22:53.221551 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.221527 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 22:22:53.230479 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.230455 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:22:53.300015 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.299929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300015 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.299968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300015 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.299990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300296 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300296 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300296 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300450 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300450 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300450 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300592 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300592 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300691 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300691 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4q5n\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300788 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300788 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300892 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300892 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300982 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.300982 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.301079 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.300998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.401618 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.401798 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.401798 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.401798 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4q5n\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.401798 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.401798 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402000 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402000 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402000 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402000 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402000 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402000 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.401977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402256 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.402003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402256 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.402025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402256 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.402050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402256 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.402076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402256 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.402098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402496 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.402462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.402983 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.402951 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.403092 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.403028 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.403092 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.403072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.403286 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.403106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.403286 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.403151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.403286 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.403242 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.403618 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.403598 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.403765 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.403743 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.403831 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.403754 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.405412 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.405386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.407414 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.407386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.409717 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.409662 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.410278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.409891 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.410278 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.410248 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.410390 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.410373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.411437 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.411275 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4q5n\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.412260 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.411789 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.412260 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.412208 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.412665 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.412646 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.413434 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.413405 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.414635 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.414602 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.417808 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.417789 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.534901 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.534634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:22:53.819270 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:53.819238 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:22:53.822895 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:22:53.822852 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14933044_aaa4_47b0_8dea_d9d407b3876b.slice/crio-328e9d0abdc49f9eecf06c6e6baebc5b7770be4a3f0ee96899df6deebf212ec1 WatchSource:0}: Error finding container 328e9d0abdc49f9eecf06c6e6baebc5b7770be4a3f0ee96899df6deebf212ec1: Status 404 returned error can't find the container with id 328e9d0abdc49f9eecf06c6e6baebc5b7770be4a3f0ee96899df6deebf212ec1 Nov 01 22:22:54.054365 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:54.054329 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerStarted","Data":"328e9d0abdc49f9eecf06c6e6baebc5b7770be4a3f0ee96899df6deebf212ec1"} Nov 01 22:22:55.059073 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:55.059038 2576 generic.go:356] "Generic (PLEG): container finished" podID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" exitCode=0 Nov 01 22:22:55.059445 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:55.059131 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerDied","Data":"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78"} Nov 01 22:22:56.065734 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:56.065695 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" event={"ID":"7059b374-3864-4296-bd7b-d77161dfca28","Type":"ContainerStarted","Data":"d82b1354cec68d08276bf1f2c83ee4bc2720f728c8fce932d374a15485730b5d"} Nov 01 22:22:56.066221 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:56.065873 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:56.072423 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:56.072394 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:22:56.119560 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:56.119506 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" podStartSLOduration=2.449070987 podStartE2EDuration="5.119489851s" totalImagesPullingTime="2.670418864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:56.094868867 +0000 UTC m=+61.996904707" watchObservedRunningTime="2026-11-01 22:22:56.119489851 +0000 UTC m=+62.021525692" Nov 01 22:22:57.444885 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.444854 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cc5c8d9fc-q5m4z"] Nov 01 22:22:57.447299 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.447278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.460744 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.460725 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc5c8d9fc-q5m4z"] Nov 01 22:22:57.541709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.541676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.541709 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.541712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.541898 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.541744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.541898 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.541761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppkjm\" (UniqueName: \"kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.541898 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.541810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.541898 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.541838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.541898 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.541861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.541898 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.541876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642187 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642187 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642385 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642385 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppkjm\" (UniqueName: \"kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642481 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642533 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642586 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642586 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642986 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642954 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.642986 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.642966 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.643199 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.643011 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.643199 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.643056 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.643359 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.643271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.644952 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.644928 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.645038 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.644969 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.650431 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.650410 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppkjm\" (UniqueName: \"kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:57.757527 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:57.757496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:22:58.045767 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:58.045743 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc5c8d9fc-q5m4z"] Nov 01 22:22:58.074248 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:58.074220 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc5c8d9fc-q5m4z" event={"ID":"ad0b4d10-a94f-444b-98b4-465e3618115a","Type":"ContainerStarted","Data":"94e7089b6c49ba7b0e70ce0f1f74fbb7993a42d8bda5427b2e144c6d9dc455f2"} Nov 01 22:22:59.079738 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.079695 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc5c8d9fc-q5m4z" event={"ID":"ad0b4d10-a94f-444b-98b4-465e3618115a","Type":"ContainerStarted","Data":"255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2"} Nov 01 22:22:59.099558 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.099500 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7cc5c8d9fc-q5m4z" podStartSLOduration=2.099482199 podStartE2EDuration="2.099482199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:59.096876475 +0000 UTC m=+64.998912315" watchObservedRunningTime="2026-11-01 22:22:59.099482199 +0000 UTC m=+65.001518042" Nov 01 22:22:59.369145 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.369056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:59.369310 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.369194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:59.371957 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.371920 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:22:59.372088 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.371998 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:22:59.382516 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.382488 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:59.382646 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.382625 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:22:59.404651 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.404603 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:59.469057 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.469025 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5scqq\"" Nov 01 22:22:59.470329 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.470304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:59.472686 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.472668 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:22:59.476940 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.476920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:22:59.479334 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.479318 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fpxpg\"" Nov 01 22:22:59.482652 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.482626 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:59.487736 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.487559 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:22:59.776864 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.776832 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9pl6q\"" Nov 01 22:22:59.788140 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.784001 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:22:59.832378 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.831479 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h5fn2"] Nov 01 22:22:59.836167 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:22:59.834598 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cgcvk"] Nov 01 22:23:00.083756 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:00.083715 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cgcvk" event={"ID":"330ca60e-5fa1-4a41-8617-73450d13cac2","Type":"ContainerStarted","Data":"9245df208253256a6b21a5a8b3acba87eeada923e0cdaeea484e1cb368163c29"} Nov 01 22:23:00.084993 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:00.084967 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h5fn2" event={"ID":"5029000f-c7ae-4f6f-9b0c-5240e385f697","Type":"ContainerStarted","Data":"cfca8b52a4784af61e7106dd99a94cd214bc4b36f3fd1d7fa530f976180f3c5e"} Nov 01 22:23:00.087091 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:00.087070 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerStarted","Data":"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc"} Nov 01 22:23:00.087218 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:00.087141 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerStarted","Data":"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7"} Nov 01 22:23:00.117216 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:00.117190 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mjw8q"] Nov 01 22:23:00.120024 ip-10-0-128-55 kubenswrapper[2576]: W1101 22:23:00.120000 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8ec790_a230_403d_8972_5634c7d2ae0f.slice/crio-429583d89724f09a7b174cc1b2bce380056dd49b60b6ed892285d631d3a74e3b WatchSource:0}: Error finding container 429583d89724f09a7b174cc1b2bce380056dd49b60b6ed892285d631d3a74e3b: Status 404 returned error can't find the container with id 429583d89724f09a7b174cc1b2bce380056dd49b60b6ed892285d631d3a74e3b Nov 01 22:23:01.093545 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:01.093502 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mjw8q" event={"ID":"7c8ec790-a230-403d-8972-5634c7d2ae0f","Type":"ContainerStarted","Data":"429583d89724f09a7b174cc1b2bce380056dd49b60b6ed892285d631d3a74e3b"} Nov 01 22:23:02.102602 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:02.102560 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mjw8q" event={"ID":"7c8ec790-a230-403d-8972-5634c7d2ae0f","Type":"ContainerStarted","Data":"29ee4d78c643c819faa3a02c199c56c0aeb608286b198c93455f06152425e97c"} Nov 01 22:23:03.110991 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:03.110948 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mjw8q" event={"ID":"7c8ec790-a230-403d-8972-5634c7d2ae0f","Type":"ContainerStarted","Data":"2297117728d6e3c6e73b8ff7821eac86e84a58c89ae131edfa6175f233b2a18b"} Nov 01 22:23:03.128886 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:03.128835 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mjw8q" podStartSLOduration=66.32487095 podStartE2EDuration="1m8.128820007s" totalImagesPullingTime="1.803949057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:03.127498023 +0000 UTC m=+69.029533865" watchObservedRunningTime="2026-11-01 22:23:03.128820007 +0000 UTC m=+69.030855846" Nov 01 22:23:05.121438 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:05.121401 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerStarted","Data":"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4"} Nov 01 22:23:05.121914 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:05.121500 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerStarted","Data":"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789"} Nov 01 22:23:05.121914 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:05.121535 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerStarted","Data":"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d"} Nov 01 22:23:05.121914 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:05.121566 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerStarted","Data":"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839"} Nov 01 22:23:05.149948 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:05.149889 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.34949704 podStartE2EDuration="12.149872331s" totalImagesPullingTime="9.783027928s" totalInitContainerRuntime="17.347363ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:05.147559674 +0000 UTC m=+71.049595518" watchObservedRunningTime="2026-11-01 22:23:05.149872331 +0000 UTC m=+71.051908172" Nov 01 22:23:07.130907 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:07.130869 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h5fn2" event={"ID":"5029000f-c7ae-4f6f-9b0c-5240e385f697","Type":"ContainerStarted","Data":"babba67fd5db0ee3a4a0edb69e02a4ee94ec5116ccef11da7c2f05091834542e"} Nov 01 22:23:07.131307 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:07.131021 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:23:07.132385 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:07.132360 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cgcvk" event={"ID":"330ca60e-5fa1-4a41-8617-73450d13cac2","Type":"ContainerStarted","Data":"4c71933ec5fb65c09d828684c335d107079436e1a7e06bdc6db623a31a0f14bd"} Nov 01 22:23:07.151684 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:07.151639 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h5fn2" podStartSLOduration=65.875441505 podStartE2EDuration="1m12.151626714s" totalImagesPullingTime="6.276185209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:07.150471221 +0000 UTC m=+73.052507061" watchObservedRunningTime="2026-11-01 22:23:07.151626714 +0000 UTC m=+73.053662570" Nov 01 22:23:07.166490 ip-10-0-128-55 kubenswrapper[2576]: I1101 22:23:07.166450 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cgcvk" podStartSLOduration=64.971103932 podStartE2EDuration="1m12.16643796s" totalImagesPullingTime="7.195334028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:07.165580524 +0000 UTC m=+73.067616364" watchObservedRunningTime="2026-11-01 22:23:07.16643796 +0000 UTC m=+73.068473800" Nov 01 22:23:07.288992 ip-10-0-128-55 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 22:23:07.303435 ip-10-0-128-55 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 22:23:07.303594 ip-10-0-128-55 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 22:23:07.303640 ip-10-0-128-55 systemd[1]: kubelet.service: Consumed 3.976s CPU time. Nov 01 22:23:07.321637 ip-10-0-128-55 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:23:07.380480 ip-10-0-128-55 kubenswrapper[6287]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:07.380480 ip-10-0-128-55 kubenswrapper[6287]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:23:07.380480 ip-10-0-128-55 kubenswrapper[6287]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:07.380480 ip-10-0-128-55 kubenswrapper[6287]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:07.382462 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382388 6287 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:07.382462 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382462 6287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:07.382462 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382465 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382469 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382471 6287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382474 6287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382477 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382480 6287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382483 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382486 6287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382489 6287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382492 6287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382500 6287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382503 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382506 6287 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382509 6287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382512 6287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382514 6287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382517 6287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382519 6287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382522 6287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:07.382559 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382524 6287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382527 6287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382529 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382532 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382535 6287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382539 6287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382543 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382546 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382549 6287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382551 6287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382554 6287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382557 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382560 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382563 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382566 6287 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382569 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382572 6287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382575 6287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382578 6287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:07.383044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382581 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382583 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382586 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382589 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382593 6287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382595 6287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382598 6287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382601 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382603 6287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382606 6287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382608 6287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382611 6287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382613 6287 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382617 6287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382622 6287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382626 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382629 6287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382632 6287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382635 6287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:07.383533 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382637 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382640 6287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382643 6287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382645 6287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382648 6287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382650 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382653 6287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382655 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382658 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382660 6287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382663 6287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382665 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382668 6287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382671 6287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382673 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382675 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382678 6287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382681 6287 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382684 6287 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382686 6287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:07.384017 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382689 6287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382691 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382694 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382696 6287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382699 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382701 6287 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382704 6287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382707 6287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382710 6287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382713 6287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382715 6287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382718 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382720 6287 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382723 6287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382726 6287 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382728 6287 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382731 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382733 6287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382736 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382739 6287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:07.384562 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382741 6287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382744 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382746 6287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382750 6287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382752 6287 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382755 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.382757 6287 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383501 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383509 6287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383512 6287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383515 6287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383518 6287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383521 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383523 6287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383526 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383529 6287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383532 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383536 6287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383540 6287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383544 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:07.385063 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383548 6287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383551 6287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383554 6287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383557 6287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383559 6287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383562 6287 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383564 6287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383567 6287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383569 6287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383572 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383575 6287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383578 6287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383582 6287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383586 6287 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383590 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383593 6287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383595 6287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383598 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383601 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383603 6287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:07.385617 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383606 6287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383609 6287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383611 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383614 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383617 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383620 6287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383622 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383625 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383628 6287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383630 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383633 6287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383636 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383639 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383641 6287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383644 6287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383647 6287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383649 6287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383652 6287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383655 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:07.386144 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383657 6287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383660 6287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383662 6287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383665 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383667 6287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383670 6287 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383673 6287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383675 6287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383678 6287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383680 6287 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383682 6287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383686 6287 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383689 6287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383691 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383694 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383696 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383699 6287 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383702 6287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383705 6287 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383708 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:07.386632 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383710 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383713 6287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383715 6287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383718 6287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383721 6287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383724 6287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383726 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383729 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383732 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383735 6287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383738 6287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383740 6287 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383743 6287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383745 6287 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383748 6287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383750 6287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383753 6287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383755 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383758 6287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383760 6287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:07.387185 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383763 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383766 6287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383768 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383771 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383774 6287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383776 6287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383779 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383781 6287 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383784 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383786 6287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383790 6287 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383792 6287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383795 6287 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.383798 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384284 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384291 6287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384294 6287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384297 6287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384301 6287 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:07.387687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384304 6287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384307 6287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384313 6287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384316 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384319 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384322 6287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384324 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384327 6287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384330 6287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384333 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384335 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384338 6287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384341 6287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384344 6287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384346 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384349 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384351 6287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384354 6287 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384357 6287 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:07.388197 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384359 6287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384362 6287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384364 6287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384367 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384370 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384372 6287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384375 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384377 6287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384380 6287 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384383 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384387 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384390 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384392 6287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384395 6287 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384398 6287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384401 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384404 6287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384407 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384409 6287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384412 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:07.388979 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384414 6287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384417 6287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384420 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384422 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384425 6287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384428 6287 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384432 6287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384435 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384439 6287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384441 6287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384444 6287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384447 6287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384450 6287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384452 6287 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384455 6287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384457 6287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384460 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384463 6287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384465 6287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:07.389754 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384468 6287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384470 6287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384473 6287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384476 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384479 6287 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384481 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384484 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384487 6287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384491 6287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384493 6287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384496 6287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384500 6287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384504 6287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384507 6287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384510 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384513 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384516 6287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384519 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384522 6287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:07.390604 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384524 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384527 6287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384530 6287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384533 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384535 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384539 6287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384541 6287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384544 6287 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384546 6287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384549 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384552 6287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384554 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384557 6287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384560 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384562 6287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384565 6287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384569 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384572 6287 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384575 6287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384577 6287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:07.391470 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384580 6287 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384585 6287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384588 6287 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.384590 6287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384680 6287 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384691 6287 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384699 6287 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384705 6287 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384712 6287 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384717 6287 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384722 6287 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384726 6287 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384729 6287 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384732 6287 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384736 6287 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384739 6287 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384742 6287 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384745 6287 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384748 6287 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384751 6287 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384754 6287 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384757 6287 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384761 6287 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384764 6287 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:23:07.392390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384767 6287 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384771 6287 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384774 6287 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384778 6287 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384782 6287 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384785 6287 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384789 6287 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384792 6287 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384795 6287 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384798 6287 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384802 6287 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384805 6287 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384809 6287 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384812 6287 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384815 6287 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384818 6287 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384821 6287 flags.go:64] FLAG: --enable-server="true" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384824 6287 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384828 6287 flags.go:64] FLAG: --event-burst="100" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384831 6287 flags.go:64] FLAG: --event-qps="50" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384834 6287 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384837 6287 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384840 6287 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384844 6287 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:23:07.393494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384847 6287 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384850 6287 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384853 6287 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384856 6287 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384859 6287 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384862 6287 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384865 6287 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384868 6287 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384871 6287 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384874 6287 flags.go:64] FLAG: --feature-gates="" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384878 6287 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384882 6287 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384887 6287 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384890 6287 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384893 6287 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384896 6287 flags.go:64] FLAG: --help="false" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384899 6287 flags.go:64] FLAG: --hostname-override="ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384903 6287 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384906 6287 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384909 6287 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384912 6287 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384915 6287 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384918 6287 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:23:07.394580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384921 6287 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384924 6287 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384927 6287 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384930 6287 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384933 6287 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384936 6287 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384939 6287 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384942 6287 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384945 6287 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384948 6287 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384951 6287 flags.go:64] FLAG: --lock-file="" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384954 6287 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384957 6287 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384959 6287 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384969 6287 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384972 6287 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384975 6287 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384978 6287 flags.go:64] FLAG: --logging-format="text" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384981 6287 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384985 6287 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384988 6287 flags.go:64] FLAG: --manifest-url="" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384992 6287 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.384997 6287 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385000 6287 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385004 6287 flags.go:64] FLAG: --max-pods="110" Nov 01 22:23:07.395641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385007 6287 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385010 6287 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385013 6287 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385016 6287 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385019 6287 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385022 6287 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385025 6287 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385031 6287 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385034 6287 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385037 6287 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385040 6287 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385043 6287 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385046 6287 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385049 6287 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385052 6287 flags.go:64] FLAG: --port="10250" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385055 6287 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385058 6287 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0dbb0c5dd5ea56a1f" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385061 6287 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385064 6287 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385068 6287 flags.go:64] FLAG: --register-node="true" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385071 6287 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385075 6287 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385078 6287 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385081 6287 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385084 6287 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385088 6287 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:23:07.396743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385091 6287 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385094 6287 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385098 6287 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385101 6287 flags.go:64] FLAG: --runonce="false" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385104 6287 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385108 6287 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385123 6287 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385126 6287 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385129 6287 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385132 6287 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385135 6287 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385138 6287 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385141 6287 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385144 6287 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385147 6287 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385150 6287 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385154 6287 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385157 6287 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385159 6287 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385165 6287 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385168 6287 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385171 6287 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385174 6287 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385177 6287 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385180 6287 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:23:07.397864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385184 6287 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385187 6287 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385190 6287 flags.go:64] FLAG: --v="2" Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385195 6287 flags.go:64] FLAG: --version="false" Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385199 6287 flags.go:64] FLAG: --vmodule="" Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385203 6287 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385206 6287 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385329 6287 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385332 6287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385335 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385339 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385341 6287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385344 6287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385346 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385349 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385352 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385355 6287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385357 6287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385360 6287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385362 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385365 6287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:07.398597 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385367 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385370 6287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385373 6287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385375 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385378 6287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385380 6287 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385383 6287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385386 6287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385388 6287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385391 6287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385393 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385397 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385400 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385403 6287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385406 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385408 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385411 6287 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385414 6287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385416 6287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385419 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:07.399205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385422 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385424 6287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385427 6287 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385430 6287 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385433 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385435 6287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385438 6287 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385440 6287 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385443 6287 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385446 6287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385448 6287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385451 6287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385453 6287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385456 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385459 6287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385461 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385464 6287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385466 6287 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385469 6287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385471 6287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:07.399862 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385474 6287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385476 6287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385479 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385483 6287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385487 6287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385491 6287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385494 6287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385497 6287 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385500 6287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385502 6287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385505 6287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385508 6287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385511 6287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385514 6287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385516 6287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385522 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385525 6287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385528 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385530 6287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:07.400415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385533 6287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385535 6287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385538 6287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385541 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385544 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385546 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385549 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385552 6287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385554 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385557 6287 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385559 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385562 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385566 6287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385570 6287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385575 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385579 6287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385585 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385590 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385595 6287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:07.400882 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385598 6287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385601 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385604 6287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385606 6287 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385609 6287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385613 6287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385616 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385619 6287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385622 6287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385624 6287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385628 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385630 6287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385633 6287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385636 6287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.385664 6287 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385701 6287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:07.401370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385706 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385708 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385712 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385714 6287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385717 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385720 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385722 6287 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385725 6287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385728 6287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385730 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385733 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385735 6287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385738 6287 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385740 6287 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385745 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385747 6287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385750 6287 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385753 6287 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385756 6287 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385758 6287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:07.401852 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385761 6287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385765 6287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385768 6287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385770 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385773 6287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385775 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385778 6287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385781 6287 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385783 6287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385786 6287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385789 6287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385791 6287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385794 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385797 6287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385801 6287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385805 6287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385808 6287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385812 6287 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385814 6287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385817 6287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:07.402500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385820 6287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385822 6287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385825 6287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385827 6287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385830 6287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385832 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385836 6287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385839 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385841 6287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385844 6287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385847 6287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385849 6287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385852 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385857 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385860 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385862 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385866 6287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385868 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385871 6287 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:07.403010 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385873 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385876 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385880 6287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385884 6287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385887 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385890 6287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385893 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385896 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385899 6287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385901 6287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385904 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385907 6287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385909 6287 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385912 6287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385914 6287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385917 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385919 6287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385922 6287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385924 6287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:07.403592 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385928 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385931 6287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385934 6287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385936 6287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385939 6287 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385942 6287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385944 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385948 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385951 6287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385953 6287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385956 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385958 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385961 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385964 6287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385967 6287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385970 6287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385973 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385975 6287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385978 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385980 6287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:07.404065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385983 6287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385985 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385988 6287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385991 6287 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385993 6287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385996 6287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.385999 6287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.391510 6287 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.391531 6287 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.391561 6287 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.391569 6287 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391656 6287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391664 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391684 6287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391689 6287 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391694 6287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391699 6287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391703 6287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391708 6287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391712 6287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:07.404665 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391717 6287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391721 6287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391726 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391730 6287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391735 6287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391739 6287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391748 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391752 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391757 6287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391762 6287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391766 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391770 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391774 6287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391779 6287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391784 6287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391789 6287 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391793 6287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391798 6287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391802 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:07.405334 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391807 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391811 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391815 6287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391820 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391824 6287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391829 6287 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391833 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391838 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391842 6287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391847 6287 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391851 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391855 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391859 6287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391863 6287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391868 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391872 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391877 6287 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391881 6287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391887 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391891 6287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:07.406057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391895 6287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391899 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391904 6287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391909 6287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391913 6287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391918 6287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391922 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391926 6287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391931 6287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391935 6287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391939 6287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391943 6287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391948 6287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391952 6287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391956 6287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391960 6287 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391965 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391969 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391974 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391978 6287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:07.406584 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391982 6287 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391986 6287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391991 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.391997 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392002 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392006 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392010 6287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392014 6287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392018 6287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392023 6287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392027 6287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392035 6287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392042 6287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392049 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392055 6287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392061 6287 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392065 6287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392072 6287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392079 6287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:07.407078 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392084 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392088 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392092 6287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392097 6287 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392101 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392105 6287 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392123 6287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392127 6287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392132 6287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392136 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392140 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392145 6287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392150 6287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392154 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392158 6287 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392163 6287 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392168 6287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392173 6287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:07.407573 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392177 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.392219 6287 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392440 6287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392448 6287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392452 6287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392457 6287 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392461 6287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392468 6287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392472 6287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392477 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392482 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392487 6287 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392491 6287 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392495 6287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392500 6287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392504 6287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:07.408005 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392509 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392513 6287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392518 6287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392522 6287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392527 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392531 6287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392535 6287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392540 6287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392544 6287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392548 6287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392552 6287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392556 6287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392561 6287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392565 6287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392571 6287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392575 6287 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392579 6287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392583 6287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392588 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392592 6287 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:07.408415 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392596 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392600 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392607 6287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392614 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392619 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392624 6287 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392628 6287 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392632 6287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392637 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392641 6287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392645 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392649 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392654 6287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392658 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392663 6287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392667 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392671 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392675 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392679 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:07.408936 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392684 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392688 6287 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392692 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392696 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392701 6287 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392705 6287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392709 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392713 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392718 6287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392722 6287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392726 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392730 6287 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392734 6287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392738 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392742 6287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392747 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392752 6287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392759 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392764 6287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:07.409488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392769 6287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392773 6287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392777 6287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392781 6287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392787 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392791 6287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392795 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392800 6287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392804 6287 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392808 6287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392812 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392816 6287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392821 6287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392825 6287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392829 6287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392833 6287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392838 6287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392842 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392846 6287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:07.409944 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392856 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392861 6287 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392865 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392869 6287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392873 6287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392878 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392883 6287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392887 6287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392891 6287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392896 6287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392900 6287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392904 6287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392908 6287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392913 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.392917 6287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:07.410414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.392954 6287 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393007 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393013 6287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393017 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393021 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393025 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393029 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393033 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393038 6287 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393042 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393047 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393051 6287 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393055 6287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393059 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393063 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393068 6287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393072 6287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393079 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393083 6287 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:07.410783 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393088 6287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393092 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393096 6287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393101 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393107 6287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393128 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393133 6287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393137 6287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393141 6287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393146 6287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393150 6287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393154 6287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393166 6287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393171 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393175 6287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393179 6287 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393183 6287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393188 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393192 6287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:07.411258 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393197 6287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393201 6287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393205 6287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393209 6287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393213 6287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393218 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393222 6287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393226 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393231 6287 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393235 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393239 6287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393246 6287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393251 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393255 6287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393259 6287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393264 6287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393268 6287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393272 6287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393277 6287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:07.411717 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393281 6287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393285 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393289 6287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393294 6287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393298 6287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393303 6287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393307 6287 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393312 6287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393316 6287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393320 6287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393325 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393331 6287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393335 6287 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393339 6287 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393343 6287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393348 6287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393352 6287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393356 6287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393361 6287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393365 6287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:07.412302 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393370 6287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393374 6287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393378 6287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393382 6287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393389 6287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393393 6287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393397 6287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393401 6287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393406 6287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393410 6287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393414 6287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393422 6287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393427 6287 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393431 6287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393436 6287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393440 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393444 6287 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393448 6287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393453 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:07.412857 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393460 6287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393468 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393473 6287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393478 6287 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393482 6287 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393486 6287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393492 6287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393496 6287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393500 6287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393504 6287 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:07.393508 6287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.393590 6287 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.395089 6287 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.395196 6287 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.395461 6287 server.go:1014] "Starting client certificate rotation" Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.395589 6287 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.395638 6287 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:16:55 +0000 UTC" deadline="2028-06-25 13:13:46.81846804 +0000 UTC" Nov 01 22:23:07.413338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.395665 6287 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14438h50m39.422804778s" Nov 01 22:23:07.413745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.396205 6287 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:07.413745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.397813 6287 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:07.413745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.399261 6287 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:23:07.413745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.401297 6287 log.go:25] "Validated CRI v1 image API" Nov 01 22:23:07.413745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.402051 6287 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:23:07.413745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.404245 6287 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81e7547f-31d2-4e8c-b370-8ede5a4c5717:/dev/nvme0n1p4 ad8bed32-968f-4a0e-8f3b-236d5f925ee2:/dev/nvme0n1p3] Nov 01 22:23:07.413936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.404265 6287 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06ea91c2eb041d2cb1012d947bb6e87ac42c33c04f87ef3a1a3add0b88ee20e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06ea91c2eb041d2cb1012d947bb6e87ac42c33c04f87ef3a1a3add0b88ee20e2/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22905f1fe76012df0956db1160fe07d3432f44a9e43b6b7d3250d9a056804d87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22905f1fe76012df0956db1160fe07d3432f44a9e43b6b7d3250d9a056804d87/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2670bb4c5609c3a48a9734677e5a4765e5bb54bbd08121afd1ba70013b636a81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2670bb4c5609c3a48a9734677e5a4765e5bb54bbd08121afd1ba70013b636a81/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/328e9d0abdc49f9eecf06c6e6baebc5b7770be4a3f0ee96899df6deebf212ec1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/328e9d0abdc49f9eecf06c6e6baebc5b7770be4a3f0ee96899df6deebf212ec1/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33099fa3a95320dbd3c785669058a2167d8d08283e0faded4365ac9df5c9b024/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33099fa3a95320dbd3c785669058a2167d8d08283e0faded4365ac9df5c9b024/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/429583d89724f09a7b174cc1b2bce380056dd49b60b6ed892285d631d3a74e3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/429583d89724f09a7b174cc1b2bce380056dd49b60b6ed892285d631d3a74e3b/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6acca285087cd2fcab46e0e1307b6b05ce41f106c393e4abb68cba8c8c2e4c46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6acca285087cd2fcab46e0e1307b6b05ce41f106c393e4abb68cba8c8c2e4c46/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7be64e7cb04ae2732b3cdaf6102ec67ccef684c704d58f4dbc68e64247f5d05a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7be64e7cb04ae2732b3cdaf6102ec67ccef684c704d58f4dbc68e64247f5d05a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9245df208253256a6b21a5a8b3acba87eeada923e0cdaeea484e1cb368163c29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9245df208253256a6b21a5a8b3acba87eeada923e0cdaeea484e1cb368163c29/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/939fd86d5f22e133516813f2ad8bca2b29b61dd22b0c5123d517a095b8594251/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/939fd86d5f22e133516813f2ad8bca2b29b61dd22b0c5123d517a095b8594251/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94e7089b6c49ba7b0e70ce0f1f74fbb7993a42d8bda5427b2e144c6d9dc455f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94e7089b6c49ba7b0e70ce0f1f74fbb7993a42d8bda5427b2e144c6d9dc455f2/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad97dc80f450d2d49b58649a2bf82779c108a3f04b7e0a80ddfda052eb7c9c46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad97dc80f450d2d49b58649a2bf82779c108a3f04b7e0a80ddfda052eb7c9c46/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0c2d46712be5b919c1e1bd150ff6236ce1c491511ac8b91ff046f63c75398fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0c2d46712be5b919c1e1bd150ff6236ce1c491511ac8b91ff046f63c75398fb/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bac89d42d44a8e9d1601d1a902dc297ac4930777254e0fd2f4ba473f2a852a57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bac89d42d44a8e9d1601d1a902dc297ac4930777254e0fd2f4ba473f2a852a57/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf754872a4c205f57cadafe91097ed7afb1912310227beb2a3a881cdf820968e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf754872a4c205f57cadafe91097ed7afb1912310227beb2a3a881cdf820968e/userdata/shm major:0 minor:313 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfca8b52a4784af61e7106dd99a94cd214bc4b36f3fd1d7fa530f976180f3c5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfca8b52a4784af61e7106dd99a94cd214bc4b36f3fd1d7fa530f976180f3c5e/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2a0c7451a9c49a26911dcf57cbae6f93fc5c63c885ae476cffdaa0b3df3fd0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2a0c7451a9c49a26911dcf57cbae6f93fc5c63c885ae476cffdaa0b3df3fd0b/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3a1dd4458820e8d696fedc27c0f89ef90ed11170e67062fe7dda115aa57cdf7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3a1dd4458820e8d696fedc27c0f89ef90ed11170e67062fe7dda115aa57cdf7/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5732415a7e4a063b587ad384d31a8bfb5d827c29650fcc73aa8a6bde1d09d18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5732415a7e4a063b587ad384d31a8bfb5d827c29650fcc73aa8a6bde1d09d18/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7ff964982721b9723e2328e469703a85263e70d75cc0db3f88f28d3040f395a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7ff964982721b9723e2328e469703a85263e70d75cc0db3f88f28d3040f395a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1608664a2958c899b08cabcc1486633d47b6a1cca5fd8000b7bbc447972f433/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1608664a2958c899b08cabcc1486633d47b6a1cca5fd8000b7bbc447972f433/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f391300cdeddeff794386ff4c2bbf9a9be6da8f6aceb6daf5cddb1c4d7a216c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f391300cdeddeff794386ff4c2bbf9a9be6da8f6aceb6daf5cddb1c4d7a216c0/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f435507ec658d780a06e7ad112fbf8f1ca1bf7380f79f5e54ce636a7407ec338/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f435507ec658d780a06e7ad112fbf8f1ca1bf7380f79f5e54ce636a7407ec338/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e214ead-da27-47f8-8196-34d885b7b4dc/volumes/kubernetes.io~projected/kube-api-access-8ww2v:{mountpoint:/var/lib/kubelet/pods/0e214ead-da27-47f8-8196-34d885b7b4dc/volumes/kubernetes.io~projected/kube-api-access-8ww2v major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volume-subpaths/web-config/config-reloader/2 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volume-subpaths/web-config/prometheus/15 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~projected/kube-api-access-w4q5n:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~projected/kube-api-access-w4q5n major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~projected/tls-assets major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/config major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/web-config major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0/volumes/kubernetes.io~projected/kube-api-access-mnvgl:{mountpoint:/var/lib/kubelet/pods/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0/volumes/kubernetes.io~projected/kube-api-access-mnvgl major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/330ca60e-5fa1-4a41-8617-73450d13cac2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/330ca60e-5fa1-4a41-8617-73450d13cac2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b137d23-cef1-4c5b-975d-3902be4f9f83/volumes/kubernetes.io~projected/kube-api-access-425j7:{mountpoint:/var/lib/kubelet/pods/3b137d23-cef1-4c5b-975d-3902be4f9f83/volumes/kubernetes.io~projected/kube-api-access-425j7 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b137d23-cef1-4c5b-975d-3902be4f9f83/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3b137d23-cef1-4c5b-975d-3902be4f9f83/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b137d23-cef1-4c5b-975d-3902be4f9f83/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/3b137d23-cef1-4c5b-975d-3902be4f9f83/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5029000f-c7ae-4f6f-9b0c-5240e385f697/volumes/kubernetes.io~projected/kube-api-access-gwb7v:{mountpoint:/var/lib/kubelet/pods/5029000f-c7ae-4f6f-9b0c-5240e385f697/volumes/kubernetes.io~projected/kube-api-access-gwb7v major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a/volumes/kubernetes.io~projected/kube-api-access-q5gjs:{mountpoint:/var/lib/kubelet/pods/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a/volumes/kubernetes.io~projected/kube-api-access-q5gjs major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/665f24e2-f8fb-4bc7-8958-191e776f78fd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/665f24e2-f8fb-4bc7-8958-191e776f78fd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/665f24e2-f8fb-4bc7-8958-191e776f78fd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/665f24e2-f8fb-4bc7-8958-191e776f78fd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/665f24e2-f8fb-4bc7-8958-191e776f78fd/volumes/kubernetes.io~projected/kube-api-access-p9q8n:{mountpoint:/var/lib/kubelet/pods/665f24e2-f8fb-4bc7-8958-191e776f78fd/volumes/kubernetes.io~projected/kube-api-access-p9q8n major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7059b374-3864-4296-bd7b-d77161dfca28/volumes/kubernetes.io~projected/kube-api-access-cfb9k:{mountpoint:/var/lib/kubelet/pods/7059b374-3864-4296-bd7b-d77161dfca28/volumes/kubernetes.io~projected/kube-api-access-cfb9k major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7059b374-3864-4296-bd7b-d77161dfca28/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/7059b374-3864-4296-bd7b-d77161dfca28/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77c13e2e-6095-4786-85d2-278dd6bf9f69/volumes/kubernetes.io~projected/kube-api-access-bb6rm:{mountpoint:/var/lib/kubelet/pods/77c13e2e-6095-4786-85d2-278dd6bf9f69/volumes/kubernetes.io~projected/kube-api-access-bb6rm major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes/kubernetes.io~projected/kube-api-access-5fvzs:{mountpoint:/var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes/kubernetes.io~projected/kube-api-access-5fvzs major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c8ec790-a230-403d-8972-5634c7d2ae0f/volumes/kubernetes.io~projected/kube-api-access-tspch:{mountpoint:/var/lib/kubelet/pods/7c8ec790-a230-403d-8972-5634c7d2ae0f/volumes/kubernetes.io~projected/kube-api-access-tspch major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c8ec790-a230-403d-8972-5634c7d2ae0f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/7c8ec790-a230-403d-8972-5634c7d2ae0f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d398858-b6ca-494f-a576-4456c4287992/volumes/kubernetes.io~projected/kube-api-access-7mdqp:{mountpoint:/var/lib/kubelet/pods/7d398858-b6ca-494f-a576-4456c4287992/volumes/kubernetes.io~projected/kube-api-access-7mdqp major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d398858-b6ca-494f-a576-4456c4287992/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7d398858-b6ca-494f-a576-4456c4287992/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d398858-b6ca-494f-a576-4456c4287992/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/7d398858-b6ca-494f-a576-4456c4287992/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/800f0c00-7f68-419a-96c5-6b947a2101a5/volumes/kubernetes.io~projected/kube-api-access-szx57:{mountpoint:/var/lib/kubelet/pods/800f0c00-7f68-419a-96c5-6b947a2101a5/volumes/kubernetes.io~projected/kube-api-access-szx57 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/800f0c00-7f68-419a-96c5-6b947a2101a5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/800f0c00-7f68-419a-96c5-6b947a2101a5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c95cb60-d629-48ef-859d-85d7893afe81/volumes/kubernetes.io~projected/kube-api-access-ljl7r:{mountpoint:/var/lib/kubelet/pods/8c95cb60-d629-48ef-859d-85d7893afe81/volumes/kubernetes.io~projected/kube-api-access-ljl7r major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c95cb60-d629-48ef-859d-85d7893afe81/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8c95cb60-d629-48ef-859d-85d7893afe81/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes/kubernetes.io~projected/kube-api-access-ppkjm:{mountpoint:/var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes/kubernetes.io~projected/kube-api-access-ppkjm major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdc57449-d2a8-41fa-808b-6ef9a3517bc6/volumes/kubernetes.io~projected/kube-api-access-tdwdz:{mountpoint:/var/lib/kubelet/pods/cdc57449-d2a8-41fa-808b-6ef9a3517bc6/volumes/kubernetes.io~projected/kube-api-access-tdwdz major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cedefae3-8d26-4f04-a6c1-78d62920c2d6/volumes/kubernetes.io~projected/kube-api-access-zpvfz:{mountpoint:/var/lib/kubelet/pods/cedefae3-8d26-4f04-a6c1-78d62920c2d6/volumes/kubernetes.io~projected/kube-api-access-zpvfz major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35d0387-f298-43a2-977a-51386c17077a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d35d0387-f298-43a2-977a-51386c17077a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35d0387-f298-43a2-977a-51386c17077a/volumes/kubernetes.io~projected/kube-api-access-svt8j:{mountpoint:/var/lib/kubelet/pods/d35d0387-f298-43a2-977a-51386c17077a/volumes/kubernetes.io~projected/kube-api-access-svt8j major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35d0387-f298-43a2-977a-51386c17077a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d35d0387-f298-43a2-977a-51386c17077a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eba18759-4d3b-468a-9a93-0044457a0b93/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/eba18759-4d3b-468a-9a93-0044457a0b93/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes/kubernetes.io~projected/kube-api-access-fprfw:{mountpoint:/var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes/kubernetes.io~projected/kube-api-access-fprfw major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1a395fe-2467-4736-9fd3-bab535023c24/volumes/kubernetes.io~projected/kube-api-access-rq4n8:{mountpoint:/var/lib/kubelet/pods/f1a395fe-2467-4736-9fd3-bab535023c24/volumes/kubernetes.io~projected/kube-api-access-rq4n8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1a395fe-2467-4736-9fd3-bab535023c24/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f1a395fe-2467-4736-9fd3-bab535023c24/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6ccfb43e57ace20f887563364d2425ff2111d0992c8a926bb96a438bfce4f423/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b3c98f2d6f6e896cdfb43613ea66b869025d0948fa6ff4f06b3868c503fde5ce/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ea3ee3ba464232dbaf377b45dafde9377d6c172f6b68965af460b17d5cfc42c8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/19be5485b7acd689fe69e75f28500b47a520f033c2d780725d771d431c677726/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/3f01fc535d3c9e4f0740db365404f738bfd6fb8a37e17f4ba9cca7e17780ffbf/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b0aeb67a79119ee153ed1374bb4947220a266a5026201efade09f8f4d3f8c6a6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/41a768124327457d7d5a9cc3c38c53ac434e47f16b1a8276ba9a475554edf96c/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9c2a936f401518a0ca94a4c669459d54ba2e26b7c53ea900b714b04bae0d2a55/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/3c2f24128640e7bc1d29d8aed3d69f79d8472ea0be6e6d146a3378d460bb1b5e/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/acdbe0aa4bf8c8801743ef9292ec80b948fdbbc6416e6aebda7ff4a5d0a1206c/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/db876c7a3855cba8c543de2b7efb534cd8f164526e3eb84520c8f72393f991ed/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/82102606b1fd9edf3de65a4d73fe3fe9831e09efd6cfbb77283de78126642498/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/88694fd0de245e8378054abf7226409c76c546eb0992f77d310bc637ff4de0d5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d619c93bb8e093cd60be7a13c7d9c550ede14244673e4f9e36c5bda18cfc7b3c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/fd2d715838a29a1f686ed6f80e15ceb1f43cdc4e7a4af9b0747b26009e293b8e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/41d2859b26289b7f8a05076794feb4cce473b070e99aec63181843d8744b0640/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/72d86ebd9c422b5415b684481897f336c4e549c33876a9c49661d2ae0a5ab938/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b127ad9f0e3e8f4b734403cf1adf816bb10cbf50bca0dff7ea27be51d5acde31/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/ac09da1a872234e3677c1186fe47e0144270e0700e68595568d0b72265a353d4/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/115d6e3d5553e6749946d0001adabd3aa5053bcb42b8f2fa2f30a9d24df4c067/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/108e95913d192c9f314d83afda6ae665466a0e5657605e7b339747485024da3d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/baba671caeb2308d357c09184894248d0c7e2dc73b573bd0f9b605b45045dcb8/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/c0dd73d51d35f742959d4844bd42d6f1fd90daa0351cede955eb93ac587f8a11/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/219f8d17cdeac191c091df821d6e3110a22772e4441cef062bb07cdfb0a4f5a5/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1f5cce23eb6294c30164dba116422191fd6f3ed23298f2618144fdf16dd34773/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/559c016ddca06a9050f10ff0ae49aba9bfe625ccaf531996528f6f3392b5c6f4/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/a31ab34849273d26d84fc31dae62ad5310091494600de0a93d383f01f26166ca/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/0e1ec97c46dc99bdefe4ca12ab4ed6af81529e503ca28a28bb85e65038e4c52a/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/968ff98e0402febe6638be895f7c9d5e4a65c7c22466fc81149fa1e8140de742/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/314702e394e21aeb72a1b475177169e4630bf18fe0b37e5c184617df416a07d0/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/f27430efe353d3b9be373c185be1932d93154b28d686319a9fce5e199462b84e/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/c07f06ec050a8f66badea7d29c65c0ded42256d01ff46fd3bb96f309177a4269/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/bf2ba22c72c2e1f280016c9b19a116ae98341552b5433a80421ef70055b0dff4/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/25e1417db41f16a9d70c1de6ac26117fa01ed93c02755604a72b52776594d5b0/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/8063cbcb545c47fb977c153926381c748331123e90e6e8690833b1dd063f3d38/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/53d0ecd18a50bd8ea729930aad2726e1d64b4e54a323ec98aa8dd6e4cafc2b1b/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/b0418593efcce955910d93bdb600cfbde7780bf89c468409601d6d3241a109bb/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/f2f012c058cfba4d7b4a8f48a2a0a192e9296aec7e08c9d6d4ead62bd091bb2e/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/448c4f5a39f19cc99ae75cea22bc83da5c74d6ed86aa219fc0a5690d28a51203/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/62c444f01ec6d03850ae1d1d35c0d30b06b210851356c68b14bc1b5479d1a5c4/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/30772e77d3572d49480355fe09dccc5512f4389a24841232d9fd14a3a215cd59/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/8a42aca072c5e9bb9b8ff45c38dc79adf4fc1622db1f88666ec7deb0e4e31a0b/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/986a8b92f171636fc6fc601e24b6006821c8cf877ae38275dc106c12b3c62b3e/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/b6c7dbb64c92d17d4e3954e8562d45d36b63a91d325c896882ee92e79702254c/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/11eb44087c1fd76637373418f37b08c81bff5017d6f63b8dbcb08cab857a43ba/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/553f73ebf218cb84a26b92d7ee49b83e3b31038bf84b314229d6fabea4fbfb01/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/2e484268311a923b8cf520460544890bacd5daa7ec3958eed440a2fc03bfa0c1/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/b3e9121891f84ed4ee285dfa2431698787c41fe1d2d2f25705ad7de4e09c43bb/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/9c2560705c4359bd34705f5b9b3f34a88385f8c579f3b15e7dc39d8c3581c92a/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/ea53c58eedb803436f75a2bee77a06c353488a640646f96b4fc592b496c1eb56/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/db5a502b6359259bcc4e2839b00920dafaa90dd5feaab4643b3d30ccc25b6aaf/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/16e66341a5932705a72fbdaffef73dd774cf5695a6af728d9b534b5f14129cba/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/1333b061e7b39cdbb63b235a95be74b305093e36cc20f51df22a3450ff13de8c/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/b6a18ac508a59082c632ffd4dac6bf5948e1a75727d1b92ee894911ccc9fc47f/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/a5e7c8dea5c4f90bde57c25c6be9b30e7f58327f8cf2abbd0ee98e71ffca5539/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/241a11ef683523a4a957b306a422ed656c8f22c738aa441b4f3289bff660a2ea/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/afbcb90ab85e9c7ab7b05a4a0cfd5ca0e0247869213d848041efb11c8896fd41/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/2374a581d915b72ca678423743042804603e51a39510256971853bef6e33480c/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/9fafaf25bf7e480fb77451dfc4a43707ec134e439fed19b9515f253d3bdd131c/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/ef15b0f1ce12872a0a50ab0d1b3c658c0f7eb25a64b019fd93b77242f4847c6c/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/61626def5d627b20c0cedf09aa48883209bb9967f0f8361cd70892d91b15abb6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/dd3d448643f982a32f6e7a4227b26f3431ae9220f2cf207d8b34809c0ae7b4de/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/137eccb8335d2ee6fc5d79ecb45efe43918e813134d5f6f7d4bf1a15c980e8f9/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/cc6d83f2eb365738f6007ca5c7c30ca68bc97ecae9376f98d401f397e81cb2df/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c398f97c959bd0768057180f242d4d71bde6ca6bf0c9756e1a88cbc6abf8e8f3/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ed1c9a500ebebeee55bceebb4cf779285eea3f011876c7e018d640ad4b4f3f5a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/853bad6391d2bc74e689addf8ee921eff2072b20953136fd8c9ad62699e7f097/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/58c958a162b443520cd30827216cc7c9f4f699cbd85c2b462812509678c9412b/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 22:23:07.414209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.413638 6287 manager.go:217] Machine: {Timestamp:2026-11-01 22:23:07.413175277 +0000 UTC m=+0.061674504 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099551 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d31d32504a7f62f2aca2ab1e08a5 SystemUUID:ec27d31d-3250-4a7f-62f2-aca2ab1e08a5 BootID:ab64b12d-0c75-484b-9df7-ef4571dc24a3 Filesystems:[{Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:318 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:323 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:325 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:320 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3a1dd4458820e8d696fedc27c0f89ef90ed11170e67062fe7dda115aa57cdf7/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes/kubernetes.io~projected/kube-api-access-5fvzs DeviceMajor:0 DeviceMinor:260 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/5029000f-c7ae-4f6f-9b0c-5240e385f697/volumes/kubernetes.io~projected/kube-api-access-gwb7v DeviceMajor:0 DeviceMinor:353 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/3b137d23-cef1-4c5b-975d-3902be4f9f83/volumes/kubernetes.io~projected/kube-api-access-425j7 DeviceMajor:0 DeviceMinor:248 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:259 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bac89d42d44a8e9d1601d1a902dc297ac4930777254e0fd2f4ba473f2a852a57/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7059b374-3864-4296-bd7b-d77161dfca28/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:308 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/eba18759-4d3b-468a-9a93-0044457a0b93/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a/volumes/kubernetes.io~projected/kube-api-access-q5gjs DeviceMajor:0 DeviceMinor:54 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1a395fe-2467-4736-9fd3-bab535023c24/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:224 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/77c13e2e-6095-4786-85d2-278dd6bf9f69/volumes/kubernetes.io~projected/kube-api-access-bb6rm DeviceMajor:0 DeviceMinor:50 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/cedefae3-8d26-4f04-a6c1-78d62920c2d6/volumes/kubernetes.io~projected/kube-api-access-zpvfz DeviceMajor:0 DeviceMinor:51 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad97dc80f450d2d49b58649a2bf82779c108a3f04b7e0a80ddfda052eb7c9c46/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7ff964982721b9723e2328e469703a85263e70d75cc0db3f88f28d3040f395a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/3b137d23-cef1-4c5b-975d-3902be4f9f83/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/800f0c00-7f68-419a-96c5-6b947a2101a5/volumes/kubernetes.io~projected/kube-api-access-szx57 DeviceMajor:0 DeviceMinor:201 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:322 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/0e214ead-da27-47f8-8196-34d885b7b4dc/volumes/kubernetes.io~projected/kube-api-access-8ww2v DeviceMajor:0 DeviceMinor:52 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9245df208253256a6b21a5a8b3acba87eeada923e0cdaeea484e1cb368163c29/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/d35d0387-f298-43a2-977a-51386c17077a/volumes/kubernetes.io~projected/kube-api-access-svt8j DeviceMajor:0 DeviceMinor:53 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/328e9d0abdc49f9eecf06c6e6baebc5b7770be4a3f0ee96899df6deebf212ec1/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2a0c7451a9c49a26911dcf57cbae6f93fc5c63c885ae476cffdaa0b3df3fd0b/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7d398858-b6ca-494f-a576-4456c4287992/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfca8b52a4784af61e7106dd99a94cd214bc4b36f3fd1d7fa530f976180f3c5e/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:335 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06ea91c2eb041d2cb1012d947bb6e87ac42c33c04f87ef3a1a3add0b88ee20e2/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/429583d89724f09a7b174cc1b2bce380056dd49b60b6ed892285d631d3a74e3b/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/800f0c00-7f68-419a-96c5-6b947a2101a5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d398858-b6ca-494f-a576-4456c4287992/volumes/kubernetes.io~projected/kube-api-access-7mdqp DeviceMajor:0 DeviceMinor:296 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/665f24e2-f8fb-4bc7-8958-191e776f78fd/volumes/kubernetes.io~projected/kube-api-access-p9q8n DeviceMajor:0 DeviceMinor:58 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c95cb60-d629-48ef-859d-85d7893afe81/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:327 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:319 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f435507ec658d780a06e7ad112fbf8f1ca1bf7380f79f5e54ce636a7407ec338/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/d35d0387-f298-43a2-977a-51386c17077a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~projected/kube-api-access-w4q5n DeviceMajor:0 DeviceMinor:326 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/d35d0387-f298-43a2-977a-51386c17077a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b137d23-cef1-4c5b-975d-3902be4f9f83/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:243 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:255 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:317 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:328 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdc57449-d2a8-41fa-808b-6ef9a3517bc6/volumes/kubernetes.io~projected/kube-api-access-tdwdz DeviceMajor:0 DeviceMinor:57 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7be64e7cb04ae2732b3cdaf6102ec67ccef684c704d58f4dbc68e64247f5d05a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:325 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes/kubernetes.io~projected/kube-api-access-ppkjm DeviceMajor:0 DeviceMinor:341 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/665f24e2-f8fb-4bc7-8958-191e776f78fd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1a395fe-2467-4736-9fd3-bab535023c24/volumes/kubernetes.io~projected/kube-api-access-rq4n8 DeviceMajor:0 DeviceMinor:202 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7c8ec790-a230-403d-8972-5634c7d2ae0f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:354 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7059b374-3864-4296-bd7b-d77161dfca28/volumes/kubernetes.io~projected/kube-api-access-cfb9k DeviceMajor:0 DeviceMinor:312 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf754872a4c205f57cadafe91097ed7afb1912310227beb2a3a881cdf820968e/userdata/shm DeviceMajor:0 DeviceMinor:313 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/939fd86d5f22e133516813f2ad8bca2b29b61dd22b0c5123d517a095b8594251/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33099fa3a95320dbd3c785669058a2167d8d08283e0faded4365ac9df5c9b024/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0c2d46712be5b919c1e1bd150ff6236ce1c491511ac8b91ff046f63c75398fb/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2670bb4c5609c3a48a9734677e5a4765e5bb54bbd08121afd1ba70013b636a81/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6acca285087cd2fcab46e0e1307b6b05ce41f106c393e4abb68cba8c8c2e4c46/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0/volumes/kubernetes.io~projected/kube-api-access-mnvgl DeviceMajor:0 DeviceMinor:49 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:321 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:324 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:325 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes/kubernetes.io~projected/kube-api-access-fprfw DeviceMajor:0 DeviceMinor:225 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:325 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94e7089b6c49ba7b0e70ce0f1f74fbb7993a42d8bda5427b2e144c6d9dc455f2/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:340 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c95cb60-d629-48ef-859d-85d7893afe81/volumes/kubernetes.io~projected/kube-api-access-ljl7r DeviceMajor:0 DeviceMinor:204 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/665f24e2-f8fb-4bc7-8958-191e776f78fd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1608664a2958c899b08cabcc1486633d47b6a1cca5fd8000b7bbc447972f433/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22905f1fe76012df0956db1160fe07d3432f44a9e43b6b7d3250d9a056804d87/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c8ec790-a230-403d-8972-5634c7d2ae0f/volumes/kubernetes.io~projected/kube-api-access-tspch DeviceMajor:0 DeviceMinor:59 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5732415a7e4a063b587ad384d31a8bfb5d827c29650fcc73aa8a6bde1d09d18/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f391300cdeddeff794386ff4c2bbf9a9be6da8f6aceb6daf5cddb1c4d7a216c0/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7d398858-b6ca-494f-a576-4456c4287992/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:295 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/330ca60e-5fa1-4a41-8617-73450d13cac2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:348 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:22905f1fe76012d MacAddress:ca:ed:5c:c1:5e:ad Speed:10000 Mtu:8901} {Name:2670bb4c5609c3a MacAddress:7e:ac:6b:74:fb:a8 Speed:10000 Mtu:8901} {Name:328e9d0abdc49f9 MacAddress:f2:70:66:e7:b0:28 Speed:10000 Mtu:8901} {Name:429583d89724f09 MacAddress:e2:d8:bd:47:a3:ab Speed:10000 Mtu:8901} {Name:6acca285087cd2f MacAddress:f2:4d:80:5f:84:cb Speed:10000 Mtu:8901} {Name:9245df208253256 MacAddress:da:d6:b0:6c:b3:b7 Speed:10000 Mtu:8901} {Name:94e7089b6c49ba7 MacAddress:ba:81:4b:bb:bc:57 Speed:10000 Mtu:8901} {Name:bf754872a4c205f MacAddress:a2:ff:12:fc:dd:60 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cf:01:92:ef Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:c3:87:d0:1c:04 Speed:0 Mtu:8901} {Name:cfca8b52a4784af MacAddress:de:55:e5:f2:69:40 Speed:10000 Mtu:8901} {Name:e2a0c7451a9c49a MacAddress:96:bc:ba:4f:9d:74 Speed:10000 Mtu:8901} {Name:e3a1dd4458820e8 MacAddress:46:8c:94:ef:54:a3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cf:01:92:ef Speed:0 Mtu:9001} {Name:f391300cdeddeff MacAddress:aa:86:82:21:c8:cf Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fa:36:64:54:bf:d7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7e:8c:0d:ea:d8:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:23:07.414209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414033 6287 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:23:07.414209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414076 6287 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:23:07.414456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414330 6287 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:23:07.414500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414347 6287 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-55.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:23:07.414547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414501 6287 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:23:07.414547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414510 6287 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:23:07.414547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414522 6287 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:07.414547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414541 6287 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:07.414652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414633 6287 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:23:07.414800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414791 6287 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:23:07.414896 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414888 6287 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:23:07.414932 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414927 6287 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:23:07.414959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414937 6287 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:23:07.414959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414948 6287 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:23:07.414959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414957 6287 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:23:07.415039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.414965 6287 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:23:07.415776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.415759 6287 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:07.415776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.415775 6287 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:07.416341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.416313 6287 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:23:07.419517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.419498 6287 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:07.419517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.419511 6287 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:07.419682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.419609 6287 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 22:23:07.419854 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.419839 6287 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:23:07.419903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.419884 6287 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:23:07.420004 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.419994 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:23:07.420034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420011 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:23:07.420034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420018 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:23:07.420034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420024 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420041 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420051 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420056 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420062 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420069 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420075 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420083 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420093 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420139 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:23:07.420175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.420149 6287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:23:07.423603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.423590 6287 server.go:1263] "Started kubelet" Nov 01 22:23:07.423810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.423643 6287 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:23:07.424219 ip-10-0-128-55 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:23:07.424813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.423726 6287 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:23:07.424813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.424660 6287 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:23:07.429829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.425512 6287 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:23:07.431616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.431045 6287 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:07.431616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.431448 6287 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:23:07.431616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.431614 6287 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:23:07.431813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.431628 6287 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:23:07.431813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.431708 6287 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:23:07.431911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.431822 6287 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:23:07.432024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.432001 6287 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:23:07.432161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.432104 6287 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:16:56 +0000 UTC" deadline="2028-08-19 19:55:28.575911914 +0000 UTC" Nov 01 22:23:07.432235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.432165 6287 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15765h32m21.143751615s" Nov 01 22:23:07.432968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.432946 6287 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:07.433893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.433869 6287 factory.go:55] Registering systemd factory Nov 01 22:23:07.433893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.433893 6287 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:23:07.434836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.434815 6287 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:07.434964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.434930 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0b4d10-a94f-444b-98b4-465e3618115a" volumeName="kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle" seLinuxMountContext="" Nov 01 22:23:07.435002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.434969 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35d0387-f298-43a2-977a-51386c17077a" volumeName="kubernetes.io/projected/d35d0387-f298-43a2-977a-51386c17077a-kube-api-access-svt8j" seLinuxMountContext="" Nov 01 22:23:07.435002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.434982 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out" seLinuxMountContext="" Nov 01 22:23:07.435002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.434996 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35d0387-f298-43a2-977a-51386c17077a" volumeName="kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-config" seLinuxMountContext="" Nov 01 22:23:07.435079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435007 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef5e5509-37bc-4415-9a55-3b85cb87511f" volumeName="kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert" seLinuxMountContext="" Nov 01 22:23:07.435079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435019 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5265e1c6-2a08-4439-b6a9-42ed2fdbd81a" volumeName="kubernetes.io/projected/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-kube-api-access-q5gjs" seLinuxMountContext="" Nov 01 22:23:07.435079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435039 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" volumeName="kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp" seLinuxMountContext="" Nov 01 22:23:07.435079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435050 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c8ec790-a230-403d-8972-5634c7d2ae0f" volumeName="kubernetes.io/projected/7c8ec790-a230-403d-8972-5634c7d2ae0f-kube-api-access-tspch" seLinuxMountContext="" Nov 01 22:23:07.435079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435065 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d398858-b6ca-494f-a576-4456c4287992" volumeName="kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:07.435079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435077 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1a395fe-2467-4736-9fd3-bab535023c24" volumeName="kubernetes.io/projected/f1a395fe-2467-4736-9fd3-bab535023c24-kube-api-access-rq4n8" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435089 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435131 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435142 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b137d23-cef1-4c5b-975d-3902be4f9f83" volumeName="kubernetes.io/configmap/3b137d23-cef1-4c5b-975d-3902be4f9f83-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435153 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5265e1c6-2a08-4439-b6a9-42ed2fdbd81a" volumeName="kubernetes.io/empty-dir/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-tmp-dir" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435178 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7059b374-3864-4296-bd7b-d77161dfca28" volumeName="kubernetes.io/projected/7059b374-3864-4296-bd7b-d77161dfca28-kube-api-access-cfb9k" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435191 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435204 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="800f0c00-7f68-419a-96c5-6b947a2101a5" volumeName="kubernetes.io/empty-dir/800f0c00-7f68-419a-96c5-6b947a2101a5-data-volume" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435215 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c95cb60-d629-48ef-859d-85d7893afe81" volumeName="kubernetes.io/secret/8c95cb60-d629-48ef-859d-85d7893afe81-metrics-tls" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435227 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0b4d10-a94f-444b-98b4-465e3618115a" volumeName="kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435239 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eba18759-4d3b-468a-9a93-0044457a0b93" volumeName="kubernetes.io/secret/eba18759-4d3b-468a-9a93-0044457a0b93-agent-certs" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435250 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435261 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435272 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b137d23-cef1-4c5b-975d-3902be4f9f83" volumeName="kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435284 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" volumeName="kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435295 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0b4d10-a94f-444b-98b4-465e3618115a" volumeName="kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435306 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0b4d10-a94f-444b-98b4-465e3618115a" volumeName="kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert" seLinuxMountContext="" Nov 01 22:23:07.435316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435317 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdc57449-d2a8-41fa-808b-6ef9a3517bc6" volumeName="kubernetes.io/configmap/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-serviceca" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435330 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef5e5509-37bc-4415-9a55-3b85cb87511f" volumeName="kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435349 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435361 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435372 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5029000f-c7ae-4f6f-9b0c-5240e385f697" volumeName="kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435384 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="665f24e2-f8fb-4bc7-8958-191e776f78fd" volumeName="kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-tmp" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435397 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77c13e2e-6095-4786-85d2-278dd6bf9f69" volumeName="kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435424 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77c13e2e-6095-4786-85d2-278dd6bf9f69" volumeName="kubernetes.io/projected/77c13e2e-6095-4786-85d2-278dd6bf9f69-kube-api-access-bb6rm" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435435 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" volumeName="kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435441 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" volumeName="kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435386 6287 factory.go:153] Registering CRI-O factory Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435472 6287 factory.go:223] Registration of the crio container factory successfully Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435517 6287 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435531 6287 factory.go:103] Registering Raw factory Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435548 6287 manager.go:1199] Started watching for new ooms in manager Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435456 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e214ead-da27-47f8-8196-34d885b7b4dc" volumeName="kubernetes.io/projected/0e214ead-da27-47f8-8196-34d885b7b4dc-kube-api-access-8ww2v" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435648 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435674 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" volumeName="kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435688 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d398858-b6ca-494f-a576-4456c4287992" volumeName="kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435702 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="800f0c00-7f68-419a-96c5-6b947a2101a5" volumeName="kubernetes.io/configmap/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435718 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c95cb60-d629-48ef-859d-85d7893afe81" volumeName="kubernetes.io/empty-dir/8c95cb60-d629-48ef-859d-85d7893afe81-tmp-dir" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435732 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c95cb60-d629-48ef-859d-85d7893afe81" volumeName="kubernetes.io/projected/8c95cb60-d629-48ef-859d-85d7893afe81-kube-api-access-ljl7r" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435745 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0b4d10-a94f-444b-98b4-465e3618115a" volumeName="kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435759 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eba18759-4d3b-468a-9a93-0044457a0b93" volumeName="kubernetes.io/configmap/eba18759-4d3b-468a-9a93-0044457a0b93-konnectivity-ca" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435771 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1a395fe-2467-4736-9fd3-bab535023c24" volumeName="kubernetes.io/secret/f1a395fe-2467-4736-9fd3-bab535023c24-cert" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435783 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435794 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435808 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435824 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" volumeName="kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435837 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7059b374-3864-4296-bd7b-d77161dfca28" volumeName="kubernetes.io/secret/7059b374-3864-4296-bd7b-d77161dfca28-monitoring-plugin-cert" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435848 6287 manager.go:319] Starting recovery of all containers Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435851 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cedefae3-8d26-4f04-a6c1-78d62920c2d6" volumeName="kubernetes.io/configmap/cedefae3-8d26-4f04-a6c1-78d62920c2d6-iptables-alerter-script" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435880 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef5e5509-37bc-4415-9a55-3b85cb87511f" volumeName="kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435896 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef5e5509-37bc-4415-9a55-3b85cb87511f" volumeName="kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435912 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435925 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435955 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0b4d10-a94f-444b-98b4-465e3618115a" volumeName="kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435979 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.435994 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436008 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436020 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" volumeName="kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436034 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0b4d10-a94f-444b-98b4-465e3618115a" volumeName="kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436046 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35d0387-f298-43a2-977a-51386c17077a" volumeName="kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-script-lib" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436060 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35d0387-f298-43a2-977a-51386c17077a" volumeName="kubernetes.io/secret/d35d0387-f298-43a2-977a-51386c17077a-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436072 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436087 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="330ca60e-5fa1-4a41-8617-73450d13cac2" volumeName="kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436099 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b137d23-cef1-4c5b-975d-3902be4f9f83" volumeName="kubernetes.io/projected/3b137d23-cef1-4c5b-975d-3902be4f9f83-kube-api-access-425j7" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436127 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="665f24e2-f8fb-4bc7-8958-191e776f78fd" volumeName="kubernetes.io/projected/665f24e2-f8fb-4bc7-8958-191e776f78fd-kube-api-access-p9q8n" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436142 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d398858-b6ca-494f-a576-4456c4287992" volumeName="kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-tls" seLinuxMountContext="" Nov 01 22:23:07.436100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436155 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdc57449-d2a8-41fa-808b-6ef9a3517bc6" volumeName="kubernetes.io/projected/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-kube-api-access-tdwdz" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436166 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35d0387-f298-43a2-977a-51386c17077a" volumeName="kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-env-overrides" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436180 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef5e5509-37bc-4415-9a55-3b85cb87511f" volumeName="kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436193 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436205 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" volumeName="kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436216 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" volumeName="kubernetes.io/projected/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-kube-api-access-mnvgl" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436227 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="665f24e2-f8fb-4bc7-8958-191e776f78fd" volumeName="kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-tuned" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436238 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" volumeName="kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436249 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="800f0c00-7f68-419a-96c5-6b947a2101a5" volumeName="kubernetes.io/projected/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-api-access-szx57" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436259 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef5e5509-37bc-4415-9a55-3b85cb87511f" volumeName="kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436272 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" volumeName="kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436283 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77c13e2e-6095-4786-85d2-278dd6bf9f69" volumeName="kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-daemon-config" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436298 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c8ec790-a230-403d-8972-5634c7d2ae0f" volumeName="kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436310 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d398858-b6ca-494f-a576-4456c4287992" volumeName="kubernetes.io/empty-dir/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-textfile" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436322 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="800f0c00-7f68-419a-96c5-6b947a2101a5" volumeName="kubernetes.io/secret/800f0c00-7f68-419a-96c5-6b947a2101a5-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436333 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c95cb60-d629-48ef-859d-85d7893afe81" volumeName="kubernetes.io/configmap/8c95cb60-d629-48ef-859d-85d7893afe81-config-volume" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436345 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cedefae3-8d26-4f04-a6c1-78d62920c2d6" volumeName="kubernetes.io/projected/cedefae3-8d26-4f04-a6c1-78d62920c2d6-kube-api-access-zpvfz" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436356 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef5e5509-37bc-4415-9a55-3b85cb87511f" volumeName="kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436367 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14933044-aaa4-47b0-8dea-d9d407b3876b" volumeName="kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436378 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b137d23-cef1-4c5b-975d-3902be4f9f83" volumeName="kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-tls" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436389 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" volumeName="kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436399 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d398858-b6ca-494f-a576-4456c4287992" volumeName="kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436412 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d398858-b6ca-494f-a576-4456c4287992" volumeName="kubernetes.io/projected/7d398858-b6ca-494f-a576-4456c4287992-kube-api-access-7mdqp" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436423 6287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0b4d10-a94f-444b-98b4-465e3618115a" volumeName="kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config" seLinuxMountContext="" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436435 6287 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:23:07.438431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.436442 6287 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:23:07.439532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.439524 6287 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 22:23:07.461626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.461578 6287 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:23:07.462923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.462897 6287 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:23:07.462923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.462924 6287 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:23:07.463100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.462945 6287 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:23:07.463100 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:07.462985 6287 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 22:23:07.467151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.467097 6287 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:07.479595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.479552 6287 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="7521b61ef36504f999e4c8ca3be403b42782024c87da483463b076d85fc09737" exitCode=0 Nov 01 22:23:07.479595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.479594 6287 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="2eda56738be3c0efa10173b8b504b3e5f65317e451c0192f3fb12c27ea25e52a" exitCode=0 Nov 01 22:23:07.479761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.479600 6287 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="5576473d3b4ceae76515a6a27b3af71cfbf821f941eaa15992a66f148d3cbb8c" exitCode=0 Nov 01 22:23:07.479761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.479607 6287 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="880795c0e9353ca6d7076cd6da0b9a2037f6ac83e1bcd5b6a082bae4e21df18f" exitCode=0 Nov 01 22:23:07.479761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.479612 6287 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="dd67ea480680b5f57832a9778364d69206aa53e221fc2eb19250f2f57809b3c8" exitCode=0 Nov 01 22:23:07.479761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.479617 6287 generic.go:356] "Generic (PLEG): container finished" podID="24c99e3c-387a-4e15-bb7c-4dc72c91a2c0" containerID="18ba5b40d83b908087c7303cc3752bcd252747d18ded2b54095c6f37f4982edf" exitCode=0 Nov 01 22:23:07.484329 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.484305 6287 generic.go:356] "Generic (PLEG): container finished" podID="cba2bbee94e76dfc255e8cf94df97bf1" containerID="98e957fbf79758a8e0ac54014bb222a78b45ce361604ed410b4ca69fdf47eb7e" exitCode=0 Nov 01 22:23:07.494636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.494610 6287 generic.go:356] "Generic (PLEG): container finished" podID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" exitCode=0 Nov 01 22:23:07.515877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.515853 6287 generic.go:356] "Generic (PLEG): container finished" podID="7d398858-b6ca-494f-a576-4456c4287992" containerID="562b6829d92dd297a61f6640f7fe4b808f1071e1c4421450b6458a6b025dfee5" exitCode=0 Nov 01 22:23:07.534944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.534925 6287 manager.go:324] Recovery completed Nov 01 22:23:07.556443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556376 6287 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:23:07.556443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556400 6287 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:23:07.556443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556422 6287 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:23:07.556655 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556620 6287 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 22:23:07.556713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556637 6287 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 22:23:07.556713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556665 6287 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 22:23:07.556713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556675 6287 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 22:23:07.556713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556683 6287 policy_none.go:50] "Start" Nov 01 22:23:07.556713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556703 6287 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:23:07.556931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.556718 6287 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:23:07.558515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.558491 6287 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 22:23:07.558515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.558510 6287 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 22:23:07.558515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.558518 6287 policy_none.go:44] "Start" Nov 01 22:23:07.563129 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:07.563097 6287 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 22:23:07.563228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.563187 6287 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:23:07.567811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.567793 6287 manager.go:341] "Starting Device Plugin manager" Nov 01 22:23:07.567889 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:07.567819 6287 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:23:07.567889 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.567829 6287 server.go:87] "Starting device plugin registration server" Nov 01 22:23:07.568499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.568481 6287 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:23:07.568568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.568494 6287 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:23:07.568616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.568575 6287 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:23:07.568701 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.568669 6287 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:23:07.568757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.568714 6287 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:23:07.568757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.568722 6287 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:23:07.568858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.568773 6287 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:07.568698464Z","UUID":"facf947f-a8e8-47fd-9ad2-ff96a584018c","Handler":null,"Name":"","Endpoint":""} Nov 01 22:23:07.570039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.570020 6287 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:23:07.570158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.570045 6287 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:23:07.570513 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:07.570490 6287 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:23:07.669317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.669289 6287 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:23:07.671218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.671200 6287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:23:07.671326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.671229 6287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:23:07.671326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.671240 6287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-55.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:23:07.671326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.671279 6287 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.682356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.682337 6287 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.682404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.682395 6287 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.763902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.763852 6287 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal"] Nov 01 22:23:07.764423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.764393 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"6ed9d7a120c3abfac5e4283e23adc9e4","Type":"ContainerStarted","Data":"ff6eed3f0f435db8701501c02704fc4bf50ec06c64035a1cabda6ecb50b3dbcb"} Nov 01 22:23:07.764500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.764485 6287 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.764557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.764486 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"6ed9d7a120c3abfac5e4283e23adc9e4","Type":"ContainerStarted","Data":"939fd86d5f22e133516813f2ad8bca2b29b61dd22b0c5123d517a095b8594251"} Nov 01 22:23:07.764557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.764521 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"cba2bbee94e76dfc255e8cf94df97bf1","Type":"ContainerStarted","Data":"53ad51a4a498b6197aa4ec3fef71a690de45cd67a346f96961ad60c5a5af0dcf"} Nov 01 22:23:07.764557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.764545 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"cba2bbee94e76dfc255e8cf94df97bf1","Type":"ContainerDied","Data":"98e957fbf79758a8e0ac54014bb222a78b45ce361604ed410b4ca69fdf47eb7e"} Nov 01 22:23:07.764702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.764565 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" event={"ID":"cba2bbee94e76dfc255e8cf94df97bf1","Type":"ContainerStarted","Data":"06ea91c2eb041d2cb1012d947bb6e87ac42c33c04f87ef3a1a3add0b88ee20e2"} Nov 01 22:23:07.764770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.764756 6287 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.777667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.777650 6287 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:07.777723 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:07.777696 6287 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.778018 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.778002 6287 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:07.778085 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:07.778042 6287 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.841391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.841330 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.841391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.841364 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ed9d7a120c3abfac5e4283e23adc9e4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"6ed9d7a120c3abfac5e4283e23adc9e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.841391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.841387 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.941642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.941615 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.941735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.941646 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.941735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.941666 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ed9d7a120c3abfac5e4283e23adc9e4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"6ed9d7a120c3abfac5e4283e23adc9e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.941735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.941722 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.941735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.941722 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cba2bbee94e76dfc255e8cf94df97bf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"cba2bbee94e76dfc255e8cf94df97bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:07.941851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:07.941797 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ed9d7a120c3abfac5e4283e23adc9e4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal\" (UID: \"6ed9d7a120c3abfac5e4283e23adc9e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:08.415444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.415408 6287 apiserver.go:52] "Watching apiserver" Nov 01 22:23:08.422692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.422669 6287 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:23:08.423278 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.423253 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8","openshift-multus/network-metrics-daemon-mjw8q","openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc","openshift-multus/multus-8rwzp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal","openshift-image-registry/node-ca-wbfqp","openshift-ingress-canary/ingress-canary-9xvw2","openshift-insights/insights-runtime-extractor-b6vcp","openshift-ovn-kubernetes/ovnkube-node-bqwkz","kube-system/global-pull-secret-syncer-cgcvk","openshift-monitoring/node-exporter-8flmn","openshift-network-operator/iptables-alerter-pchz7","kube-system/konnectivity-agent-prbmb","openshift-network-diagnostics/network-check-target-h5fn2","openshift-cluster-node-tuning-operator/tuned-zlskb","openshift-monitoring/prometheus-k8s-0","openshift-dns/dns-default-78mfv","openshift-multus/multus-additional-cni-plugins-s86mt","openshift-console/console-59557fbb88-l56pg","openshift-console/console-7cc5c8d9fc-q5m4z","openshift-dns/node-resolver-9pqxl","openshift-console/console-56fb5c6985-4cddf"] Nov 01 22:23:08.427334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.427309 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.427500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.427452 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.427688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.427669 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.427785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.427704 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.427840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.427829 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:23:08.428370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.428338 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.428466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.428389 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:23:08.428466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.428389 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hpp8\"" Nov 01 22:23:08.428466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.428395 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:23:08.428639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.428499 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-79k59\"" Nov 01 22:23:08.428639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.428532 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:23:08.428856 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.428819 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.428933 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.428874 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:23:08.429008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.428994 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.429074 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.429027 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.429140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.429071 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.429140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.429086 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.429612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.429590 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.430462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.430436 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:23:08.431322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.431303 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:23:08.431547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.431524 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:23:08.431907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.431871 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.431986 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.431920 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fpxpg\"" Nov 01 22:23:08.432041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432006 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p26bd\"" Nov 01 22:23:08.432160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.431402 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.432338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432309 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:23:08.432415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432381 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.432555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432512 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dclh4\"" Nov 01 22:23:08.432663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432647 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:23:08.432754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.431308 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j5zd8\"" Nov 01 22:23:08.432868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432849 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5scqq\"" Nov 01 22:23:08.432930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432910 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:23:08.432977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432948 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:08.433028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432778 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:23:08.433139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432752 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:08.433246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433137 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:23:08.433356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433334 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vvcmq\"" Nov 01 22:23:08.433438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.432756 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4kglz\"" Nov 01 22:23:08.433540 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433446 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qjk9v\"" Nov 01 22:23:08.433641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433457 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.433733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433536 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.433815 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433749 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4bt99\"" Nov 01 22:23:08.433963 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433774 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.434057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433821 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.434177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433859 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9pl6q\"" Nov 01 22:23:08.434261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.433964 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:23:08.434380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.434361 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zwl5x\"" Nov 01 22:23:08.434439 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.434263 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vv5lb\"" Nov 01 22:23:08.434439 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.434101 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:23:08.434532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.434429 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:23:08.434893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.434847 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:23:08.443484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443459 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-system-cni-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.443566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443497 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-kubernetes\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.443612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443560 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-run\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.443612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443586 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c95cb60-d629-48ef-859d-85d7893afe81-metrics-tls\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:23:08.443676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443626 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-lib-modules\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.443676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443660 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvfz\" (UniqueName: \"kubernetes.io/projected/cedefae3-8d26-4f04-a6c1-78d62920c2d6-kube-api-access-zpvfz\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:23:08.443757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443699 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.443757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443730 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvgl\" (UniqueName: \"kubernetes.io/projected/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-kube-api-access-mnvgl\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.443850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443755 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eba18759-4d3b-468a-9a93-0044457a0b93-agent-certs\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:23:08.443850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443772 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.443850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443787 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-config\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.443850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443806 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svt8j\" (UniqueName: \"kubernetes.io/projected/d35d0387-f298-43a2-977a-51386c17077a-kube-api-access-svt8j\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.443850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443830 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9q8n\" (UniqueName: \"kubernetes.io/projected/665f24e2-f8fb-4bc7-8958-191e776f78fd-kube-api-access-p9q8n\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.443850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443842 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c95cb60-d629-48ef-859d-85d7893afe81-metrics-tls\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443852 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443876 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443904 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-binary-copy\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443902 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-system-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443953 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-bin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.443996 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-daemon-config\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444021 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-etc-kubernetes\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444042 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-kubelet\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444071 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-var-lib-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444071 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eba18759-4d3b-468a-9a93-0044457a0b93-agent-certs\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444098 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-config\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444132 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-kubelet-config\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444100 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.444170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444170 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cedefae3-8d26-4f04-a6c1-78d62920c2d6-host-slash\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444209 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-hostroot\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444241 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-conf-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444271 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-daemon-config\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444278 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cnibin\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444312 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tspch\" (UniqueName: \"kubernetes.io/projected/7c8ec790-a230-403d-8972-5634c7d2ae0f-kube-api-access-tspch\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444338 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-netns\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444361 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-node-log\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444401 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444444 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444470 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444508 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-systemd-units\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444533 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-log-socket\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.444654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444556 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-netd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.445032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444673 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwdz\" (UniqueName: \"kubernetes.io/projected/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-kube-api-access-tdwdz\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:23:08.445032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444709 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:23:08.445032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444710 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.445032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444735 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1a395fe-2467-4736-9fd3-bab535023c24-cert\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:23:08.445032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444902 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.445032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.444983 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8ec790-a230-403d-8972-5634c7d2ae0f-metrics-certs\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:23:08.445255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445145 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6rm\" (UniqueName: \"kubernetes.io/projected/77c13e2e-6095-4786-85d2-278dd6bf9f69-kube-api-access-bb6rm\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.445255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445197 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.445327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445308 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-sys-fs\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.445360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445337 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-bin\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.445401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445371 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d35d0387-f298-43a2-977a-51386c17077a-ovn-node-metrics-cert\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.445401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445388 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.445468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445403 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.445468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445429 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.445611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445579 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-modprobe-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.445708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445623 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-host\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.445708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445672 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d35d0387-f298-43a2-977a-51386c17077a-ovn-node-metrics-cert\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.445789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445709 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gjs\" (UniqueName: \"kubernetes.io/projected/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-kube-api-access-q5gjs\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:23:08.445789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445715 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.445789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445749 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-multus\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.445789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445771 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-kubelet\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.445953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445861 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/800f0c00-7f68-419a-96c5-6b947a2101a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.445953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445941 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-serviceca\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:23:08.446042 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.445978 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-systemd\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.446098 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446086 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-hosts-file\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:23:08.446257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446101 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/800f0c00-7f68-419a-96c5-6b947a2101a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.446257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446123 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysconfig\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.446257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446175 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-sys\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.446257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446194 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-k8s-cni-cncf-io\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.446257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446241 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95cb60-d629-48ef-859d-85d7893afe81-tmp-dir\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:23:08.446430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446261 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szx57\" (UniqueName: \"kubernetes.io/projected/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-api-access-szx57\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.446430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446284 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eba18759-4d3b-468a-9a93-0044457a0b93-konnectivity-ca\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:23:08.446430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446300 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-slash\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.446430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446314 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-var-lib-kubelet\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.446430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446318 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95cb60-d629-48ef-859d-85d7893afe81-tmp-dir\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:23:08.446430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446330 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-tuned\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.446430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446371 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446497 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-netns\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446539 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-tuned\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446539 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446551 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eba18759-4d3b-468a-9a93-0044457a0b93-konnectivity-ca\") pod \"konnectivity-agent-prbmb\" (UID: \"eba18759-4d3b-468a-9a93-0044457a0b93\") " pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446587 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446601 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-serviceca\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446613 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-ovn\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446650 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-conf\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446678 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-tmp-dir\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446694 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.446729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446710 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprfw\" (UniqueName: \"kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446755 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446755 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-tmp-dir\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446769 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-cnibin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446878 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446919 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.446983 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-tmp\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447011 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447027 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-socket-dir-parent\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447042 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-registration-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447058 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-dbus\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447094 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4n8\" (UniqueName: \"kubernetes.io/projected/f1a395fe-2467-4736-9fd3-bab535023c24-kube-api-access-rq4n8\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:23:08.447152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447141 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c95cb60-d629-48ef-859d-85d7893afe81-config-volume\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447164 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-systemd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447179 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-script-lib\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447194 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-cni-binary-copy\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447221 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljl7r\" (UniqueName: \"kubernetes.io/projected/8c95cb60-d629-48ef-859d-85d7893afe81-kube-api-access-ljl7r\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447249 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/330ca60e-5fa1-4a41-8617-73450d13cac2-original-pull-secret\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447296 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/665f24e2-f8fb-4bc7-8958-191e776f78fd-tmp\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447332 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/800f0c00-7f68-419a-96c5-6b947a2101a5-crio-socket\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447409 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-device-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447450 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ww2v\" (UniqueName: \"kubernetes.io/projected/0e214ead-da27-47f8-8196-34d885b7b4dc-kube-api-access-8ww2v\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447485 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77c13e2e-6095-4786-85d2-278dd6bf9f69-cni-binary-copy\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447497 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c95cb60-d629-48ef-859d-85d7893afe81-config-volume\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447512 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-ovnkube-script-lib\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447521 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-etc-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447571 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-env-overrides\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447595 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-host\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447621 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cedefae3-8d26-4f04-a6c1-78d62920c2d6-iptables-alerter-script\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:23:08.447649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447646 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.448271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447674 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-multus-certs\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.448271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447693 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d35d0387-f298-43a2-977a-51386c17077a-env-overrides\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.448271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447768 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/800f0c00-7f68-419a-96c5-6b947a2101a5-data-volume\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.448271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447787 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-socket-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.448271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447836 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/800f0c00-7f68-419a-96c5-6b947a2101a5-data-volume\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.448271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.447845 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cedefae3-8d26-4f04-a6c1-78d62920c2d6-iptables-alerter-script\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:23:08.451818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.451794 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvgl\" (UniqueName: \"kubernetes.io/projected/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-kube-api-access-mnvgl\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.451894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.451821 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svt8j\" (UniqueName: \"kubernetes.io/projected/d35d0387-f298-43a2-977a-51386c17077a-kube-api-access-svt8j\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.452082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.452062 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvfz\" (UniqueName: \"kubernetes.io/projected/cedefae3-8d26-4f04-a6c1-78d62920c2d6-kube-api-access-zpvfz\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:23:08.452440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.452425 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwb7v\" (UniqueName: \"kubernetes.io/projected/5029000f-c7ae-4f6f-9b0c-5240e385f697-kube-api-access-gwb7v\") pod \"network-check-target-h5fn2\" (UID: \"5029000f-c7ae-4f6f-9b0c-5240e385f697\") " pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:23:08.452474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.452426 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9q8n\" (UniqueName: \"kubernetes.io/projected/665f24e2-f8fb-4bc7-8958-191e776f78fd-kube-api-access-p9q8n\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.453359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.453340 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tspch\" (UniqueName: \"kubernetes.io/projected/7c8ec790-a230-403d-8972-5634c7d2ae0f-kube-api-access-tspch\") pod \"network-metrics-daemon-mjw8q\" (UID: \"7c8ec790-a230-403d-8972-5634c7d2ae0f\") " pod="openshift-multus/network-metrics-daemon-mjw8q" Nov 01 22:23:08.453450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.453412 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gjs\" (UniqueName: \"kubernetes.io/projected/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-kube-api-access-q5gjs\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:23:08.453450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.453412 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwdz\" (UniqueName: \"kubernetes.io/projected/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-kube-api-access-tdwdz\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:23:08.457155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.457132 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szx57\" (UniqueName: \"kubernetes.io/projected/800f0c00-7f68-419a-96c5-6b947a2101a5-kube-api-access-szx57\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.457292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.457278 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljl7r\" (UniqueName: \"kubernetes.io/projected/8c95cb60-d629-48ef-859d-85d7893afe81-kube-api-access-ljl7r\") pod \"dns-default-78mfv\" (UID: \"8c95cb60-d629-48ef-859d-85d7893afe81\") " pod="openshift-dns/dns-default-78mfv" Nov 01 22:23:08.457292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.457282 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6rm\" (UniqueName: \"kubernetes.io/projected/77c13e2e-6095-4786-85d2-278dd6bf9f69-kube-api-access-bb6rm\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.457354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.457327 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ww2v\" (UniqueName: \"kubernetes.io/projected/0e214ead-da27-47f8-8196-34d885b7b4dc-kube-api-access-8ww2v\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.521486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.521460 6287 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:08.521643 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.521619 6287 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:08.527753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.527727 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 22:23:08.527753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.527738 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 22:23:08.527943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.527815 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 22:23:08.527943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.527817 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.527943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.527825 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 22:23:08.528089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.527951 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 22:23:08.528089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.527990 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.528089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.528063 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.528733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.528697 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 22:23:08.528944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.528915 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.529175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.529154 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 22:23:08.529285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.529268 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 22:23:08.529341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.529325 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 22:23:08.529387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.529375 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 22:23:08.530959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.530749 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 22:23:08.531595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.531575 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 22:23:08.532056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.532038 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 22:23:08.534139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.534090 6287 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:08.534243 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:08.534168 6287 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:08.534578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.534554 6287 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:08.534648 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:08.534626 6287 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal" Nov 01 22:23:08.534849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.534830 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:23:08.534927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.534859 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4fw2s\"" Nov 01 22:23:08.534927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.534872 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.535031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.534951 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 22:23:08.535217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.535200 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1a395fe-2467-4736-9fd3-bab535023c24-cert\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:23:08.535280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.535242 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 22:23:08.535605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.535579 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 22:23:08.535713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.535640 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 22:23:08.535781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.535716 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2acjdgdf71rol\"" Nov 01 22:23:08.535781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.535768 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 22:23:08.535876 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.535822 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 22:23:08.536030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536011 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 22:23:08.536030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536018 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 22:23:08.536263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536106 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-w2wlc\"" Nov 01 22:23:08.536263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536137 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 22:23:08.536263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536174 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 22:23:08.536263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536201 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 22:23:08.536263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536210 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 22:23:08.536263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536246 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zj5nt\"" Nov 01 22:23:08.536648 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536627 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mjnqw\"" Nov 01 22:23:08.536736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536680 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4jwn7\"" Nov 01 22:23:08.536736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536701 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 22:23:08.536843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536788 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 22:23:08.536843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536794 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.536937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536888 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h46mq\"" Nov 01 22:23:08.536937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536925 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 22:23:08.537032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.536962 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 22:23:08.538067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.538044 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprfw\" (UniqueName: \"kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.540195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.538175 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4n8\" (UniqueName: \"kubernetes.io/projected/f1a395fe-2467-4736-9fd3-bab535023c24-kube-api-access-rq4n8\") pod \"ingress-canary-9xvw2\" (UID: \"f1a395fe-2467-4736-9fd3-bab535023c24\") " pod="openshift-ingress-canary/ingress-canary-9xvw2" Nov 01 22:23:08.540195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.538250 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 22:23:08.540195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.538292 6287 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:23:08.540195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.538395 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.540195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.539222 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.545941 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.545917 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert\") pod \"console-56fb5c6985-4cddf\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:08.547992 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.547970 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-host\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.548081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.547997 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdqp\" (UniqueName: \"kubernetes.io/projected/7d398858-b6ca-494f-a576-4456c4287992-kube-api-access-7mdqp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.548081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548014 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-kubelet\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.548081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548030 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.548081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548045 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.548081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548064 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548081 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-host\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548102 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-kubelet\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548087 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-slash\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548159 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-slash\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548229 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-var-lib-kubelet\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548272 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-var-lib-kubelet\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548314 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425j7\" (UniqueName: \"kubernetes.io/projected/3b137d23-cef1-4c5b-975d-3902be4f9f83-kube-api-access-425j7\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548338 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548370 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.548379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548376 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548403 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-netns\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548426 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548450 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-ovn\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548490 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-netns\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548497 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-ovn\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548515 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-conf\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548529 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548541 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548569 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7059b374-3864-4296-bd7b-d77161dfca28-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548599 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548632 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-conf\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548637 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548663 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548735 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548737 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548775 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548789 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548808 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7059b374-3864-4296-bd7b-d77161dfca28-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548819 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfb9k\" (UniqueName: \"kubernetes.io/projected/7059b374-3864-4296-bd7b-d77161dfca28-kube-api-access-cfb9k\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:23:08.548865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548852 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548899 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-dbus\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548926 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-run\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548950 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-device-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.548986 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549003 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-multus-certs\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549051 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-socket-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549098 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549145 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-wtmp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549150 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-multus-certs\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549180 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549214 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549097 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-run\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549252 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-device-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549300 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549331 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-socket-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549353 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549390 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549420 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-system-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549429 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549432 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549445 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-kubelet\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549485 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-var-lib-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549496 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-kubelet\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549517 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-node-log\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549550 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549567 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-system-cni-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549580 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549610 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-kubelet-config\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549642 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cedefae3-8d26-4f04-a6c1-78d62920c2d6-host-slash\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549648 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549678 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-netns\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549693 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549711 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-modprobe-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549750 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-var-lib-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549779 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-tls\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549807 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cedefae3-8d26-4f04-a6c1-78d62920c2d6-host-slash\") pod \"iptables-alerter-pchz7\" (UID: \"cedefae3-8d26-4f04-a6c1-78d62920c2d6\") " pod="openshift-network-operator/iptables-alerter-pchz7" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549820 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-systemd-units\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549824 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.549870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549875 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-textfile\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549909 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-node-log\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549914 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b137d23-cef1-4c5b-975d-3902be4f9f83-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549947 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-metrics-client-ca\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549960 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-kubelet-config\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549976 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550022 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550033 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550070 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550103 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550146 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-multus\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550170 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-systemd\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550222 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550233 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-tls\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550260 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-accelerators-collector-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550299 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.549486 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550321 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-textfile\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550331 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-hosts-file\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550236 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b137d23-cef1-4c5b-975d-3902be4f9f83-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550364 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysconfig\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550390 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-sys\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550423 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-k8s-cni-cncf-io\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550456 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550481 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550506 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550523 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-metrics-client-ca\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550529 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-cnibin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550528 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550525 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550556 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550578 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b137d23-cef1-4c5b-975d-3902be4f9f83-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550582 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550594 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550625 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550639 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550644 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-systemd-units\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550653 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-socket-dir-parent\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550661 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-systemd\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550701 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-registration-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550704 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-socket-dir-parent\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550719 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550731 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-sys\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550755 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysconfig\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550764 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550758 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550798 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550763 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-multus\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550812 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550829 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-accelerators-collector-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550830 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550869 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-modprobe-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550301 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-run-netns\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550924 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550953 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550957 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5265e1c6-2a08-4439-b6a9-42ed2fdbd81a-hosts-file\") pod \"node-resolver-9pqxl\" (UID: \"5265e1c6-2a08-4439-b6a9-42ed2fdbd81a\") " pod="openshift-dns/node-resolver-9pqxl" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.550960 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-sys\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551005 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551018 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-registration-dir\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551022 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551020 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-cnibin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551034 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551064 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551066 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/330ca60e-5fa1-4a41-8617-73450d13cac2-dbus\") pod \"global-pull-secret-syncer-cgcvk\" (UID: \"330ca60e-5fa1-4a41-8617-73450d13cac2\") " pod="kube-system/global-pull-secret-syncer-cgcvk" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551076 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-run-k8s-cni-cncf-io\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551087 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551100 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-systemd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551142 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551153 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551175 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-run-systemd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551184 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551195 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551273 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551275 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/800f0c00-7f68-419a-96c5-6b947a2101a5-crio-socket\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551300 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551322 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/800f0c00-7f68-419a-96c5-6b947a2101a5-crio-socket\") pod \"insights-runtime-extractor-b6vcp\" (UID: \"800f0c00-7f68-419a-96c5-6b947a2101a5\") " pod="openshift-insights/insights-runtime-extractor-b6vcp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551321 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-etc-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551342 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551351 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-host\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551353 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-etc-openvswitch\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551380 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-system-cni-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551407 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-system-cni-dir\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551407 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc57449-d2a8-41fa-808b-6ef9a3517bc6-host\") pod \"node-ca-wbfqp\" (UID: \"cdc57449-d2a8-41fa-808b-6ef9a3517bc6\") " pod="openshift-image-registry/node-ca-wbfqp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551419 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-kubernetes\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551435 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-lib-modules\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551452 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551454 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-kubernetes\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551467 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551482 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551497 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551544 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-lib-modules\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551574 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4q5n\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551603 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551633 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-bin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551655 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551635 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551663 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-etc-kubernetes\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551691 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-host-var-lib-cni-bin\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551695 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551714 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-etc-kubernetes\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551684 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551735 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-bin\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551711 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-bin\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.551684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551760 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551785 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvzs\" (UniqueName: \"kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551810 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-hostroot\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551832 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-conf-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551855 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cnibin\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551878 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-log-socket\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551890 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-hostroot\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551853 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/665f24e2-f8fb-4bc7-8958-191e776f78fd-etc-sysctl-d\") pod \"tuned-zlskb\" (UID: \"665f24e2-f8fb-4bc7-8958-191e776f78fd\") " pod="openshift-cluster-node-tuning-operator/tuned-zlskb" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551902 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-root\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551911 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77c13e2e-6095-4786-85d2-278dd6bf9f69-multus-conf-dir\") pod \"multus-8rwzp\" (UID: \"77c13e2e-6095-4786-85d2-278dd6bf9f69\") " pod="openshift-multus/multus-8rwzp" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551894 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24c99e3c-387a-4e15-bb7c-4dc72c91a2c0-cnibin\") pod \"multus-additional-cni-plugins-s86mt\" (UID: \"24c99e3c-387a-4e15-bb7c-4dc72c91a2c0\") " pod="openshift-multus/multus-additional-cni-plugins-s86mt" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551952 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-log-socket\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551944 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551962 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.551985 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppkjm\" (UniqueName: \"kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.552012 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-netd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.552034 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.552079 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d35d0387-f298-43a2-977a-51386c17077a-host-cni-netd\") pod \"ovnkube-node-bqwkz\" (UID: \"d35d0387-f298-43a2-977a-51386c17077a\") " pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.552097 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.552132 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.552173 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-sys-fs\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.552180 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.552245 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.552305 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e214ead-da27-47f8-8196-34d885b7b4dc-sys-fs\") pod \"aws-ebs-csi-driver-node-9kxlv\" (UID: \"0e214ead-da27-47f8-8196-34d885b7b4dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.553276 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59557fbb88-l56pg"] Nov 01 22:23:08.555727 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:08.553517 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5fvzs], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-console/console-59557fbb88-l56pg" podUID="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" Nov 01 22:23:08.557327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.557306 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-425j7\" (UniqueName: \"kubernetes.io/projected/3b137d23-cef1-4c5b-975d-3902be4f9f83-kube-api-access-425j7\") pod \"prometheus-operator-9cd89b74c-tpkmc\" (UID: \"3b137d23-cef1-4c5b-975d-3902be4f9f83\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" Nov 01 22:23:08.557434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.557407 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfb9k\" (UniqueName: \"kubernetes.io/projected/7059b374-3864-4296-bd7b-d77161dfca28-kube-api-access-cfb9k\") pod \"monitoring-plugin-77f4f76594-xqjq8\" (UID: \"7059b374-3864-4296-bd7b-d77161dfca28\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:23:08.557550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.557534 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdqp\" (UniqueName: \"kubernetes.io/projected/7d398858-b6ca-494f-a576-4456c4287992-kube-api-access-7mdqp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.564919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.564896 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppkjm\" (UniqueName: \"kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm\") pod \"console-7cc5c8d9fc-q5m4z\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.565008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.564980 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4q5n\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n\") pod \"prometheus-k8s-0\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:08.565347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.565329 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvzs\" (UniqueName: \"kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs\") pod \"console-59557fbb88-l56pg\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:08.652512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.652486 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-wtmp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.652647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.652535 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-sys\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.652647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.652569 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-root\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.652647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.652602 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-node-exporter-wtmp\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.652740 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.652660 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-root\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.652740 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.652660 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d398858-b6ca-494f-a576-4456c4287992-sys\") pod \"node-exporter-8flmn\" (UID: \"7d398858-b6ca-494f-a576-4456c4287992\") " pod="openshift-monitoring/node-exporter-8flmn" Nov 01 22:23:08.792702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.792674 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:08.803764 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:08.803740 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-78mfv" Nov 01 22:23:09.547222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:09.547195 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:09.665865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:09.665835 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:09.680682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:09.680655 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:11.661044 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:11.661012 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:14.904305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:14.904265 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:23:15.585994 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:15.585960 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:23:17.244944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:17.244857 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:17.245522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:17.245175 6287 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:23:18.610243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:18.610213 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:23:18.798059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:18.798029 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:18.801762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:18.801739 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:23:20.982798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:20.982774 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" Nov 01 22:23:28.327659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.327626 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fdc5c4886-h5qt9"] Nov 01 22:23:28.334855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.334830 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.341439 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.341412 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdc5c4886-h5qt9"] Nov 01 22:23:28.405778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.405746 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-tmp\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.405778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.405781 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-trusted-ca-bundle\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.406012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.405930 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-console-serving-cert\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.406012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.405987 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-oauth-serving-cert\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.406144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.406021 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-service-ca\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.406144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.406137 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-console-oauth-config\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.406261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.406187 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2k9\" (UniqueName: \"kubernetes.io/projected/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-kube-api-access-kz2k9\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.406261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.406217 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-console-config\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.507447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.507401 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2k9\" (UniqueName: \"kubernetes.io/projected/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-kube-api-access-kz2k9\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.507447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.507451 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-console-config\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.507682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.507471 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-tmp\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.507682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.507489 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-trusted-ca-bundle\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.507682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.507529 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-console-serving-cert\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.507682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.507563 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-oauth-serving-cert\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.507682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.507589 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-service-ca\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.507682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.507633 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-console-oauth-config\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.508060 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.508023 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-tmp\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.508429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.508408 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-console-config\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.508520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.508407 6287 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:23:28.509050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.509023 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-oauth-serving-cert\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.509275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.509254 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-service-ca\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.509362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.509251 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-trusted-ca-bundle\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.510555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.510537 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-console-oauth-config\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.510819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.510802 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-console-serving-cert\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.515074 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.515054 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2k9\" (UniqueName: \"kubernetes.io/projected/49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26-kube-api-access-kz2k9\") pod \"console-5fdc5c4886-h5qt9\" (UID: \"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26\") " pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.647295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.647216 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:28.919982 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:28.919907 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdc5c4886-h5qt9"] Nov 01 22:23:28.922846 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:28.922816 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e9bbfa_4014_4bd1_8ba1_f00d8fa0da26.slice/crio-52006fe5ec3598ce22357ed1faa9d331f0709c5760138c4fe7888504303622fb WatchSource:0}: Error finding container 52006fe5ec3598ce22357ed1faa9d331f0709c5760138c4fe7888504303622fb: Status 404 returned error can't find the container with id 52006fe5ec3598ce22357ed1faa9d331f0709c5760138c4fe7888504303622fb Nov 01 22:23:29.592391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:29.592349 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdc5c4886-h5qt9" event={"ID":"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26","Type":"ContainerStarted","Data":"8104eda1e836d47fcee07a737130da6159dfcb9a01b21c13706a26b8b43dafca"} Nov 01 22:23:29.593016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:29.592414 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdc5c4886-h5qt9" event={"ID":"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26","Type":"ContainerStarted","Data":"52006fe5ec3598ce22357ed1faa9d331f0709c5760138c4fe7888504303622fb"} Nov 01 22:23:33.455980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.455941 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-56fb5c6985-4cddf" podUID="ef5e5509-37bc-4415-9a55-3b85cb87511f" containerName="console" containerID="cri-o://e8cf9dda765cb973156e028af260a3afb507ee05e012fe02afa01ccb5083d3e4" gracePeriod=15 Nov 01 22:23:33.609272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.609239 6287 generic.go:356] "Generic (PLEG): container finished" podID="ef5e5509-37bc-4415-9a55-3b85cb87511f" containerID="e8cf9dda765cb973156e028af260a3afb507ee05e012fe02afa01ccb5083d3e4" exitCode=0 Nov 01 22:23:33.609470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.609298 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fb5c6985-4cddf" event={"ID":"ef5e5509-37bc-4415-9a55-3b85cb87511f","Type":"ContainerDied","Data":"e8cf9dda765cb973156e028af260a3afb507ee05e012fe02afa01ccb5083d3e4"} Nov 01 22:23:33.704499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.704474 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:33.723414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.723312 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fdc5c4886-h5qt9" podStartSLOduration=5.723293539 podStartE2EDuration="5.723293539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:29.610797445 +0000 UTC m=+22.259296680" watchObservedRunningTime="2026-11-01 22:23:33.723293539 +0000 UTC m=+26.371792774" Nov 01 22:23:33.856505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856468 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca\") pod \"ef5e5509-37bc-4415-9a55-3b85cb87511f\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " Nov 01 22:23:33.856672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856540 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert\") pod \"ef5e5509-37bc-4415-9a55-3b85cb87511f\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " Nov 01 22:23:33.856672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856577 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config\") pod \"ef5e5509-37bc-4415-9a55-3b85cb87511f\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " Nov 01 22:23:33.856672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856598 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw\" (UniqueName: \"kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw\") pod \"ef5e5509-37bc-4415-9a55-3b85cb87511f\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " Nov 01 22:23:33.856672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856644 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp\") pod \"ef5e5509-37bc-4415-9a55-3b85cb87511f\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " Nov 01 22:23:33.856913 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856693 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config\") pod \"ef5e5509-37bc-4415-9a55-3b85cb87511f\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " Nov 01 22:23:33.856913 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856720 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert\") pod \"ef5e5509-37bc-4415-9a55-3b85cb87511f\" (UID: \"ef5e5509-37bc-4415-9a55-3b85cb87511f\") " Nov 01 22:23:33.857012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856930 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp" pod "ef5e5509-37bc-4415-9a55-3b85cb87511f" (UID: "ef5e5509-37bc-4415-9a55-3b85cb87511f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:33.857012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856970 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert" pod "ef5e5509-37bc-4415-9a55-3b85cb87511f" (UID: "ef5e5509-37bc-4415-9a55-3b85cb87511f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:33.857012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.856934 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca" pod "ef5e5509-37bc-4415-9a55-3b85cb87511f" (UID: "ef5e5509-37bc-4415-9a55-3b85cb87511f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:33.857174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.857151 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config" pod "ef5e5509-37bc-4415-9a55-3b85cb87511f" (UID: "ef5e5509-37bc-4415-9a55-3b85cb87511f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:33.859026 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.858995 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config" pod "ef5e5509-37bc-4415-9a55-3b85cb87511f" (UID: "ef5e5509-37bc-4415-9a55-3b85cb87511f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:33.859338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.859061 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert" pod "ef5e5509-37bc-4415-9a55-3b85cb87511f" (UID: "ef5e5509-37bc-4415-9a55-3b85cb87511f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:33.859338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.859068 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw" pod "ef5e5509-37bc-4415-9a55-3b85cb87511f" (UID: "ef5e5509-37bc-4415-9a55-3b85cb87511f"). InnerVolumeSpecName "kube-api-access-fprfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:33.920855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.920826 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:33.957571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.957433 6287 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-config\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:33.957571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.957467 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fprfw\" (UniqueName: \"kubernetes.io/projected/ef5e5509-37bc-4415-9a55-3b85cb87511f-kube-api-access-fprfw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:33.957571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.957483 6287 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef5e5509-37bc-4415-9a55-3b85cb87511f-tmp\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:33.957571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.957498 6287 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-oauth-config\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:33.957571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.957512 6287 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef5e5509-37bc-4415-9a55-3b85cb87511f-console-serving-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:33.957571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.957525 6287 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:33.957571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.957538 6287 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef5e5509-37bc-4415-9a55-3b85cb87511f-oauth-serving-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:33.963057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:33.963026 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:23:34.544350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.544288 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59557fbb88-l56pg" podUID="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" containerName="console" containerID="cri-o://4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe" gracePeriod=15 Nov 01 22:23:34.614085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.614052 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fb5c6985-4cddf" Nov 01 22:23:34.614255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.614085 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fb5c6985-4cddf" event={"ID":"ef5e5509-37bc-4415-9a55-3b85cb87511f","Type":"ContainerDied","Data":"f391300cdeddeff794386ff4c2bbf9a9be6da8f6aceb6daf5cddb1c4d7a216c0"} Nov 01 22:23:34.614255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.614155 6287 scope.go:122] "RemoveContainer" containerID="e8cf9dda765cb973156e028af260a3afb507ee05e012fe02afa01ccb5083d3e4" Nov 01 22:23:34.614454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.614429 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="prometheus" containerID="cri-o://8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" gracePeriod=600 Nov 01 22:23:34.614649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.614623 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy" containerID="cri-o://d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" gracePeriod=600 Nov 01 22:23:34.614758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.614728 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy-web" containerID="cri-o://3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" gracePeriod=600 Nov 01 22:23:34.614819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.614800 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="config-reloader" containerID="cri-o://2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" gracePeriod=600 Nov 01 22:23:34.614944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.614744 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" gracePeriod=600 Nov 01 22:23:34.614944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.614890 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="thanos-sidecar" containerID="cri-o://93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" gracePeriod=600 Nov 01 22:23:34.660930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.660904 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56fb5c6985-4cddf"] Nov 01 22:23:34.664322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.664302 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56fb5c6985-4cddf"] Nov 01 22:23:34.838661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.838636 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:34.897590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.897565 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:34.964610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.964584 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca\") pod \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " Nov 01 22:23:34.964746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.964633 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs\" (UniqueName: \"kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs\") pod \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " Nov 01 22:23:34.964746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.964675 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert\") pod \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " Nov 01 22:23:34.964746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.964735 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle\") pod \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " Nov 01 22:23:34.964886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.964775 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp\") pod \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " Nov 01 22:23:34.964886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.964808 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config\") pod \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " Nov 01 22:23:34.964886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.964864 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config\") pod \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " Nov 01 22:23:34.965033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.964908 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert\") pod \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\" (UID: \"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b\") " Nov 01 22:23:34.965400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.965106 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp" pod "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" (UID: "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:34.965400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.964999 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca" pod "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" (UID: "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:34.965400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.965208 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert" pod "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" (UID: "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:34.965400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.965345 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle" pod "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" (UID: "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:34.965603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.965413 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config" pod "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" (UID: "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:34.966954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.966934 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs" pod "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" (UID: "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b"). InnerVolumeSpecName "kube-api-access-5fvzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:34.967043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.966953 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config" pod "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" (UID: "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:34.967043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:34.967009 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert" pod "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" (UID: "7c6f0fd2-0f11-4cc4-abe2-169f81abd94b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.066377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066316 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066360 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066382 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066406 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066423 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066449 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066469 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066487 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066507 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066540 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066568 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.066604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066592 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.067038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066626 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.067038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066660 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.067038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066689 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.067038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066722 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.067038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066746 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:35.067038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066757 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.067038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066820 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.067038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066863 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.067038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.066896 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out\") pod \"14933044-aaa4-47b0-8dea-d9d407b3876b\" (UID: \"14933044-aaa4-47b0-8dea-d9d407b3876b\") " Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067135 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067178 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067219 6287 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-config\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067239 6287 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-serving-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067255 6287 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067271 6287 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067286 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fvzs\" (UniqueName: \"kubernetes.io/projected/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-kube-api-access-5fvzs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067301 6287 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-oauth-serving-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067315 6287 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-trusted-ca-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067330 6287 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067345 6287 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-tmp\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.067513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067359 6287 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b-console-oauth-config\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.068135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.067522 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:35.068135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.068062 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:35.069175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.069147 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.069285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.069197 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "kube-api-access-w4q5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:35.069628 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.069603 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:23:35.069749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.069721 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:35.070032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.070004 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.070578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.070421 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.070900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.070759 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.070900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.070841 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:35.071184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.071146 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:23:35.071273 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.071208 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.071516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.071496 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.071776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.071753 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.071913 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.071895 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:23:35.072221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.072199 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.079379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.079362 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config" pod "14933044-aaa4-47b0-8dea-d9d407b3876b" (UID: "14933044-aaa4-47b0-8dea-d9d407b3876b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:23:35.167826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167804 6287 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167825 6287 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167835 6287 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167845 6287 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-config-out\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167855 6287 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167864 6287 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-db\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167873 6287 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167882 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4q5n\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-kube-api-access-w4q5n\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167891 6287 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167899 6287 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-metrics-client-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167908 6287 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-web-config\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167917 6287 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-grpc-tls\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167925 6287 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167934 6287 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-config\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167942 6287 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14933044-aaa4-47b0-8dea-d9d407b3876b-tls-assets\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.167954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167951 6287 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.168395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167960 6287 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14933044-aaa4-47b0-8dea-d9d407b3876b-secret-kube-rbac-proxy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.168395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.167969 6287 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14933044-aaa4-47b0-8dea-d9d407b3876b-configmap-metrics-client-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:23:35.468275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.468204 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5e5509-37bc-4415-9a55-3b85cb87511f" path="/var/lib/kubelet/pods/ef5e5509-37bc-4415-9a55-3b85cb87511f/volumes" Nov 01 22:23:35.623177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623147 6287 generic.go:356] "Generic (PLEG): container finished" podID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerID="525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" exitCode=0 Nov 01 22:23:35.623177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623175 6287 generic.go:356] "Generic (PLEG): container finished" podID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerID="d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" exitCode=0 Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623184 6287 generic.go:356] "Generic (PLEG): container finished" podID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerID="3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" exitCode=0 Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623190 6287 generic.go:356] "Generic (PLEG): container finished" podID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerID="93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" exitCode=0 Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623195 6287 generic.go:356] "Generic (PLEG): container finished" podID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerID="2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" exitCode=0 Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623202 6287 generic.go:356] "Generic (PLEG): container finished" podID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerID="8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" exitCode=0 Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623232 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerDied","Data":"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4"} Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623245 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623360 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerDied","Data":"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789"} Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623407 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerDied","Data":"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d"} Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623443 6287 scope.go:122] "RemoveContainer" containerID="525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623447 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerDied","Data":"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839"} Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623589 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerDied","Data":"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc"} Nov 01 22:23:35.623645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623630 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerDied","Data":"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7"} Nov 01 22:23:35.624313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.623670 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14933044-aaa4-47b0-8dea-d9d407b3876b","Type":"ContainerDied","Data":"328e9d0abdc49f9eecf06c6e6baebc5b7770be4a3f0ee96899df6deebf212ec1"} Nov 01 22:23:35.625215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.625190 6287 generic.go:356] "Generic (PLEG): container finished" podID="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" containerID="4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe" exitCode=0 Nov 01 22:23:35.625315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.625264 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59557fbb88-l56pg" Nov 01 22:23:35.625315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.625284 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59557fbb88-l56pg" event={"ID":"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b","Type":"ContainerDied","Data":"4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe"} Nov 01 22:23:35.625413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.625332 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59557fbb88-l56pg" event={"ID":"7c6f0fd2-0f11-4cc4-abe2-169f81abd94b","Type":"ContainerDied","Data":"e2a0c7451a9c49a26911dcf57cbae6f93fc5c63c885ae476cffdaa0b3df3fd0b"} Nov 01 22:23:35.631316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.631300 6287 scope.go:122] "RemoveContainer" containerID="d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" Nov 01 22:23:35.637629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.637612 6287 scope.go:122] "RemoveContainer" containerID="3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" Nov 01 22:23:35.641750 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.641732 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59557fbb88-l56pg"] Nov 01 22:23:35.644396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.644381 6287 scope.go:122] "RemoveContainer" containerID="93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" Nov 01 22:23:35.645606 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.645587 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59557fbb88-l56pg"] Nov 01 22:23:35.650819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.650804 6287 scope.go:122] "RemoveContainer" containerID="2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" Nov 01 22:23:35.656878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.656860 6287 scope.go:122] "RemoveContainer" containerID="8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" Nov 01 22:23:35.659786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.659763 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:23:35.664011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.663969 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:23:35.665369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.665347 6287 scope.go:122] "RemoveContainer" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" Nov 01 22:23:35.671699 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.671681 6287 scope.go:122] "RemoveContainer" containerID="525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" Nov 01 22:23:35.671954 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:35.671933 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": container with ID starting with 525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4 not found: ID does not exist" containerID="525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" Nov 01 22:23:35.672000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.671962 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4"} err="failed to get container status \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": rpc error: code = NotFound desc = could not find container \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": container with ID starting with 525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4 not found: ID does not exist" Nov 01 22:23:35.672048 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.672003 6287 scope.go:122] "RemoveContainer" containerID="d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" Nov 01 22:23:35.672260 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:35.672241 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": container with ID starting with d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789 not found: ID does not exist" containerID="d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" Nov 01 22:23:35.672308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.672265 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789"} err="failed to get container status \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": rpc error: code = NotFound desc = could not find container \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": container with ID starting with d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789 not found: ID does not exist" Nov 01 22:23:35.672308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.672280 6287 scope.go:122] "RemoveContainer" containerID="3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" Nov 01 22:23:35.672499 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:35.672482 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": container with ID starting with 3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d not found: ID does not exist" containerID="3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" Nov 01 22:23:35.672542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.672504 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d"} err="failed to get container status \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": rpc error: code = NotFound desc = could not find container \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": container with ID starting with 3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d not found: ID does not exist" Nov 01 22:23:35.672542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.672518 6287 scope.go:122] "RemoveContainer" containerID="93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" Nov 01 22:23:35.672749 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:35.672732 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": container with ID starting with 93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839 not found: ID does not exist" containerID="93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" Nov 01 22:23:35.672787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.672755 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839"} err="failed to get container status \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": rpc error: code = NotFound desc = could not find container \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": container with ID starting with 93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839 not found: ID does not exist" Nov 01 22:23:35.672787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.672768 6287 scope.go:122] "RemoveContainer" containerID="2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" Nov 01 22:23:35.672969 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:35.672956 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": container with ID starting with 2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc not found: ID does not exist" containerID="2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" Nov 01 22:23:35.673015 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.672972 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc"} err="failed to get container status \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": rpc error: code = NotFound desc = could not find container \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": container with ID starting with 2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc not found: ID does not exist" Nov 01 22:23:35.673015 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.672983 6287 scope.go:122] "RemoveContainer" containerID="8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" Nov 01 22:23:35.673232 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:35.673214 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": container with ID starting with 8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7 not found: ID does not exist" containerID="8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" Nov 01 22:23:35.673272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.673237 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7"} err="failed to get container status \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": rpc error: code = NotFound desc = could not find container \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": container with ID starting with 8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7 not found: ID does not exist" Nov 01 22:23:35.673272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.673251 6287 scope.go:122] "RemoveContainer" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" Nov 01 22:23:35.673476 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:35.673460 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": container with ID starting with 2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78 not found: ID does not exist" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" Nov 01 22:23:35.673516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.673478 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78"} err="failed to get container status \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": rpc error: code = NotFound desc = could not find container \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": container with ID starting with 2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78 not found: ID does not exist" Nov 01 22:23:35.673516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.673489 6287 scope.go:122] "RemoveContainer" containerID="525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" Nov 01 22:23:35.673716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.673700 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4"} err="failed to get container status \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": rpc error: code = NotFound desc = could not find container \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": container with ID starting with 525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4 not found: ID does not exist" Nov 01 22:23:35.673754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.673716 6287 scope.go:122] "RemoveContainer" containerID="d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" Nov 01 22:23:35.673910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.673892 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789"} err="failed to get container status \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": rpc error: code = NotFound desc = could not find container \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": container with ID starting with d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789 not found: ID does not exist" Nov 01 22:23:35.673947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.673911 6287 scope.go:122] "RemoveContainer" containerID="3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" Nov 01 22:23:35.674143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.674102 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d"} err="failed to get container status \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": rpc error: code = NotFound desc = could not find container \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": container with ID starting with 3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d not found: ID does not exist" Nov 01 22:23:35.674210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.674145 6287 scope.go:122] "RemoveContainer" containerID="93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" Nov 01 22:23:35.674363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.674346 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839"} err="failed to get container status \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": rpc error: code = NotFound desc = could not find container \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": container with ID starting with 93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839 not found: ID does not exist" Nov 01 22:23:35.674431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.674364 6287 scope.go:122] "RemoveContainer" containerID="2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" Nov 01 22:23:35.674601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.674583 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc"} err="failed to get container status \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": rpc error: code = NotFound desc = could not find container \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": container with ID starting with 2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc not found: ID does not exist" Nov 01 22:23:35.674650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.674602 6287 scope.go:122] "RemoveContainer" containerID="8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" Nov 01 22:23:35.674801 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.674786 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7"} err="failed to get container status \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": rpc error: code = NotFound desc = could not find container \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": container with ID starting with 8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7 not found: ID does not exist" Nov 01 22:23:35.674854 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.674802 6287 scope.go:122] "RemoveContainer" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" Nov 01 22:23:35.675011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.674993 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78"} err="failed to get container status \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": rpc error: code = NotFound desc = could not find container \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": container with ID starting with 2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78 not found: ID does not exist" Nov 01 22:23:35.675131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.675012 6287 scope.go:122] "RemoveContainer" containerID="525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" Nov 01 22:23:35.675243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.675226 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4"} err="failed to get container status \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": rpc error: code = NotFound desc = could not find container \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": container with ID starting with 525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4 not found: ID does not exist" Nov 01 22:23:35.675287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.675244 6287 scope.go:122] "RemoveContainer" containerID="d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" Nov 01 22:23:35.675430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.675409 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789"} err="failed to get container status \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": rpc error: code = NotFound desc = could not find container \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": container with ID starting with d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789 not found: ID does not exist" Nov 01 22:23:35.675467 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.675431 6287 scope.go:122] "RemoveContainer" containerID="3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" Nov 01 22:23:35.675627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.675609 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d"} err="failed to get container status \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": rpc error: code = NotFound desc = could not find container \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": container with ID starting with 3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d not found: ID does not exist" Nov 01 22:23:35.675700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.675629 6287 scope.go:122] "RemoveContainer" containerID="93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" Nov 01 22:23:35.675849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.675830 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839"} err="failed to get container status \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": rpc error: code = NotFound desc = could not find container \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": container with ID starting with 93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839 not found: ID does not exist" Nov 01 22:23:35.675896 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.675850 6287 scope.go:122] "RemoveContainer" containerID="2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" Nov 01 22:23:35.676056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676036 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc"} err="failed to get container status \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": rpc error: code = NotFound desc = could not find container \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": container with ID starting with 2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc not found: ID does not exist" Nov 01 22:23:35.676192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676058 6287 scope.go:122] "RemoveContainer" containerID="8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" Nov 01 22:23:35.676306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676287 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7"} err="failed to get container status \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": rpc error: code = NotFound desc = could not find container \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": container with ID starting with 8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7 not found: ID does not exist" Nov 01 22:23:35.676353 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676307 6287 scope.go:122] "RemoveContainer" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" Nov 01 22:23:35.676531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676513 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78"} err="failed to get container status \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": rpc error: code = NotFound desc = could not find container \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": container with ID starting with 2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78 not found: ID does not exist" Nov 01 22:23:35.676579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676533 6287 scope.go:122] "RemoveContainer" containerID="525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" Nov 01 22:23:35.676726 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676709 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4"} err="failed to get container status \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": rpc error: code = NotFound desc = could not find container \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": container with ID starting with 525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4 not found: ID does not exist" Nov 01 22:23:35.676770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676726 6287 scope.go:122] "RemoveContainer" containerID="d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" Nov 01 22:23:35.676908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676892 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789"} err="failed to get container status \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": rpc error: code = NotFound desc = could not find container \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": container with ID starting with d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789 not found: ID does not exist" Nov 01 22:23:35.676953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.676909 6287 scope.go:122] "RemoveContainer" containerID="3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" Nov 01 22:23:35.677089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677074 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d"} err="failed to get container status \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": rpc error: code = NotFound desc = could not find container \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": container with ID starting with 3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d not found: ID does not exist" Nov 01 22:23:35.677089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677088 6287 scope.go:122] "RemoveContainer" containerID="93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" Nov 01 22:23:35.677296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677277 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839"} err="failed to get container status \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": rpc error: code = NotFound desc = could not find container \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": container with ID starting with 93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839 not found: ID does not exist" Nov 01 22:23:35.677364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677297 6287 scope.go:122] "RemoveContainer" containerID="2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" Nov 01 22:23:35.677509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677494 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc"} err="failed to get container status \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": rpc error: code = NotFound desc = could not find container \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": container with ID starting with 2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc not found: ID does not exist" Nov 01 22:23:35.677550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677509 6287 scope.go:122] "RemoveContainer" containerID="8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" Nov 01 22:23:35.677692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677675 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7"} err="failed to get container status \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": rpc error: code = NotFound desc = could not find container \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": container with ID starting with 8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7 not found: ID does not exist" Nov 01 22:23:35.677756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677693 6287 scope.go:122] "RemoveContainer" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" Nov 01 22:23:35.677927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677907 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78"} err="failed to get container status \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": rpc error: code = NotFound desc = could not find container \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": container with ID starting with 2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78 not found: ID does not exist" Nov 01 22:23:35.677974 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.677928 6287 scope.go:122] "RemoveContainer" containerID="525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" Nov 01 22:23:35.678146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.678131 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4"} err="failed to get container status \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": rpc error: code = NotFound desc = could not find container \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": container with ID starting with 525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4 not found: ID does not exist" Nov 01 22:23:35.678182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.678145 6287 scope.go:122] "RemoveContainer" containerID="d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" Nov 01 22:23:35.678371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.678350 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789"} err="failed to get container status \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": rpc error: code = NotFound desc = could not find container \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": container with ID starting with d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789 not found: ID does not exist" Nov 01 22:23:35.678406 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.678373 6287 scope.go:122] "RemoveContainer" containerID="3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" Nov 01 22:23:35.678569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.678549 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d"} err="failed to get container status \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": rpc error: code = NotFound desc = could not find container \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": container with ID starting with 3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d not found: ID does not exist" Nov 01 22:23:35.678612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.678570 6287 scope.go:122] "RemoveContainer" containerID="93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" Nov 01 22:23:35.678810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.678792 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839"} err="failed to get container status \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": rpc error: code = NotFound desc = could not find container \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": container with ID starting with 93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839 not found: ID does not exist" Nov 01 22:23:35.678858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.678810 6287 scope.go:122] "RemoveContainer" containerID="2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" Nov 01 22:23:35.679005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.678981 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc"} err="failed to get container status \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": rpc error: code = NotFound desc = could not find container \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": container with ID starting with 2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc not found: ID does not exist" Nov 01 22:23:35.679054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.679005 6287 scope.go:122] "RemoveContainer" containerID="8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" Nov 01 22:23:35.679277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.679260 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7"} err="failed to get container status \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": rpc error: code = NotFound desc = could not find container \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": container with ID starting with 8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7 not found: ID does not exist" Nov 01 22:23:35.679277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.679277 6287 scope.go:122] "RemoveContainer" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" Nov 01 22:23:35.679492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.679473 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78"} err="failed to get container status \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": rpc error: code = NotFound desc = could not find container \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": container with ID starting with 2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78 not found: ID does not exist" Nov 01 22:23:35.679535 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.679493 6287 scope.go:122] "RemoveContainer" containerID="525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4" Nov 01 22:23:35.679692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.679672 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4"} err="failed to get container status \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": rpc error: code = NotFound desc = could not find container \"525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4\": container with ID starting with 525a8550cf71a5a106289db4981f892eac379b344d91febe1d834f4cafb358b4 not found: ID does not exist" Nov 01 22:23:35.679732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.679692 6287 scope.go:122] "RemoveContainer" containerID="d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789" Nov 01 22:23:35.679899 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.679880 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789"} err="failed to get container status \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": rpc error: code = NotFound desc = could not find container \"d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789\": container with ID starting with d7eb8da47f8e516618f150d0b8f8b555d8066441ef3659246313055af747f789 not found: ID does not exist" Nov 01 22:23:35.679936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.679899 6287 scope.go:122] "RemoveContainer" containerID="3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d" Nov 01 22:23:35.680102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680084 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d"} err="failed to get container status \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": rpc error: code = NotFound desc = could not find container \"3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d\": container with ID starting with 3290dc3c0da7ca5383788627770cac0022f68ac69482a8be19a299c0f57c2b8d not found: ID does not exist" Nov 01 22:23:35.680154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680102 6287 scope.go:122] "RemoveContainer" containerID="93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839" Nov 01 22:23:35.680310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680293 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839"} err="failed to get container status \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": rpc error: code = NotFound desc = could not find container \"93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839\": container with ID starting with 93751e763c8017ed16e6bab6ee5e74378cbb00fbb09084f1a688c1b4be314839 not found: ID does not exist" Nov 01 22:23:35.680349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680310 6287 scope.go:122] "RemoveContainer" containerID="2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc" Nov 01 22:23:35.680525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680509 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc"} err="failed to get container status \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": rpc error: code = NotFound desc = could not find container \"2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc\": container with ID starting with 2db42595359e066ded0dcf71a05afbf8d27e7be9c17c19c8c8cbfae9ec340fbc not found: ID does not exist" Nov 01 22:23:35.680580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680524 6287 scope.go:122] "RemoveContainer" containerID="8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7" Nov 01 22:23:35.680733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680717 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7"} err="failed to get container status \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": rpc error: code = NotFound desc = could not find container \"8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7\": container with ID starting with 8b97fe386c12039f70c40719dd0f9b614a7a444550df7d0ff58a303db81470d7 not found: ID does not exist" Nov 01 22:23:35.680733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680731 6287 scope.go:122] "RemoveContainer" containerID="2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78" Nov 01 22:23:35.680912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680898 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78"} err="failed to get container status \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": rpc error: code = NotFound desc = could not find container \"2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78\": container with ID starting with 2ab28f85badd71abfd79bc4a2848ffb2187830c5a24f6b32425435405da78a78 not found: ID does not exist" Nov 01 22:23:35.680912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.680911 6287 scope.go:122] "RemoveContainer" containerID="4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe" Nov 01 22:23:35.687427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.687413 6287 scope.go:122] "RemoveContainer" containerID="4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe" Nov 01 22:23:35.687685 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:35.687668 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe\": container with ID starting with 4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe not found: ID does not exist" containerID="4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe" Nov 01 22:23:35.687725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.687690 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe"} err="failed to get container status \"4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe\": rpc error: code = NotFound desc = could not find container \"4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe\": container with ID starting with 4e1e280160f6eb5305c4b49d3a096bdbebd6a923e0b10e2198a8c85b55f959fe not found: ID does not exist" Nov 01 22:23:35.694653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694633 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:23:35.694897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694883 6287 cpu_manager.go:414] "containerMap: removing container" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="init-config-reloader" Nov 01 22:23:35.694961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694900 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="init-config-reloader" Nov 01 22:23:35.694961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694919 6287 cpu_manager.go:414] "containerMap: removing container" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy-web" Nov 01 22:23:35.694961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694927 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy-web" Nov 01 22:23:35.694961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694943 6287 cpu_manager.go:414] "containerMap: removing container" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy" Nov 01 22:23:35.694961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694951 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy" Nov 01 22:23:35.694961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694961 6287 cpu_manager.go:414] "containerMap: removing container" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="prometheus" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694968 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="prometheus" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.694996 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ef5e5509-37bc-4415-9a55-3b85cb87511f" containerName="console" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695004 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5e5509-37bc-4415-9a55-3b85cb87511f" containerName="console" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695022 6287 cpu_manager.go:414] "containerMap: removing container" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy-thanos" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695028 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy-thanos" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695040 6287 cpu_manager.go:414] "containerMap: removing container" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="thanos-sidecar" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695047 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="thanos-sidecar" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695056 6287 cpu_manager.go:414] "containerMap: removing container" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="config-reloader" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695064 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="config-reloader" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695075 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" containerName="console" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695082 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" containerName="console" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695173 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695186 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="prometheus" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695198 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5e5509-37bc-4415-9a55-3b85cb87511f" containerName="console" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695208 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy-thanos" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695216 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="thanos-sidecar" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695223 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="config-reloader" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695232 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" containerName="console" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695240 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="init-config-reloader" Nov 01 22:23:35.695298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.695249 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" containerName="kube-rbac-proxy-web" Nov 01 22:23:35.700256 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.700238 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.704718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.704702 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 22:23:35.705157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705028 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4fw2s\"" Nov 01 22:23:35.705157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705097 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705358 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705367 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705365 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705434 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705546 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705622 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705640 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705675 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705705 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705721 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 22:23:35.706900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.705729 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 22:23:35.707805 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.707786 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2acjdgdf71rol\"" Nov 01 22:23:35.707943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.707926 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 22:23:35.708158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.708143 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 22:23:35.719318 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.719274 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:23:35.873022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.872984 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-config\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873037 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873109 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873155 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873178 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-web-config\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873195 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873314 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873332 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873353 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873371 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97258e3b-22d2-487b-932c-d8dd9432d372-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873407 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97258e3b-22d2-487b-932c-d8dd9432d372-config-out\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873461 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873506 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873530 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873549 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873578 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873618 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873643 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873659 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrlk\" (UniqueName: \"kubernetes.io/projected/97258e3b-22d2-487b-932c-d8dd9432d372-kube-api-access-rkrlk\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.873786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.873713 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974201 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974237 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974251 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974268 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-web-config\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974433 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974403 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974468 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974492 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974562 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974522 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974562 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974549 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97258e3b-22d2-487b-932c-d8dd9432d372-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974573 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97258e3b-22d2-487b-932c-d8dd9432d372-config-out\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974606 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974633 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974658 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974682 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974710 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974734 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974760 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.974804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974784 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrlk\" (UniqueName: \"kubernetes.io/projected/97258e3b-22d2-487b-932c-d8dd9432d372-kube-api-access-rkrlk\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.975100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974814 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.975100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.974858 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-config\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.975100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.975047 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.975100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.975059 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.975317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.975209 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.976254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.975854 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.976254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.975987 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.976439 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.976256 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.977951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.977922 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-web-config\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.978956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.978923 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.979218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.979199 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.979352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.979207 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.979449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.979368 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-config\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.979542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.979476 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.979605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.979531 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97258e3b-22d2-487b-932c-d8dd9432d372-config-out\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.979605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.979571 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.979788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.979766 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.980240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.980211 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.980304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.980239 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/97258e3b-22d2-487b-932c-d8dd9432d372-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.980519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.980497 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/97258e3b-22d2-487b-932c-d8dd9432d372-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.981076 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.981061 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97258e3b-22d2-487b-932c-d8dd9432d372-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:35.984129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:35.984088 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrlk\" (UniqueName: \"kubernetes.io/projected/97258e3b-22d2-487b-932c-d8dd9432d372-kube-api-access-rkrlk\") pod \"prometheus-k8s-0\" (UID: \"97258e3b-22d2-487b-932c-d8dd9432d372\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:36.025095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:36.025074 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:36.244723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:36.244696 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-prbmb" Nov 01 22:23:36.311610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:36.311580 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:23:36.314983 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:23:36.314958 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97258e3b_22d2_487b_932c_d8dd9432d372.slice/crio-8046e7101cd2108dd26caa09148d732e44ff411b93bc2f3cedc6177ee9c6f083 WatchSource:0}: Error finding container 8046e7101cd2108dd26caa09148d732e44ff411b93bc2f3cedc6177ee9c6f083: Status 404 returned error can't find the container with id 8046e7101cd2108dd26caa09148d732e44ff411b93bc2f3cedc6177ee9c6f083 Nov 01 22:23:36.632548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:36.632464 6287 generic.go:356] "Generic (PLEG): container finished" podID="97258e3b-22d2-487b-932c-d8dd9432d372" containerID="88b83e5be745bc95cbdc8a75d2428fe5d595e5bdc0ff9a131a591fd798e74eaf" exitCode=0 Nov 01 22:23:36.632905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:36.632550 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97258e3b-22d2-487b-932c-d8dd9432d372","Type":"ContainerDied","Data":"88b83e5be745bc95cbdc8a75d2428fe5d595e5bdc0ff9a131a591fd798e74eaf"} Nov 01 22:23:36.632905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:36.632641 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97258e3b-22d2-487b-932c-d8dd9432d372","Type":"ContainerStarted","Data":"8046e7101cd2108dd26caa09148d732e44ff411b93bc2f3cedc6177ee9c6f083"} Nov 01 22:23:37.467924 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:37.467891 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14933044-aaa4-47b0-8dea-d9d407b3876b" path="/var/lib/kubelet/pods/14933044-aaa4-47b0-8dea-d9d407b3876b/volumes" Nov 01 22:23:37.468574 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:37.468558 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6f0fd2-0f11-4cc4-abe2-169f81abd94b" path="/var/lib/kubelet/pods/7c6f0fd2-0f11-4cc4-abe2-169f81abd94b/volumes" Nov 01 22:23:37.639499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:37.639465 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97258e3b-22d2-487b-932c-d8dd9432d372","Type":"ContainerStarted","Data":"9982fa845f175c3d93ed369f608d6085a832b93e1f7f5a614171d4af267469ab"} Nov 01 22:23:37.639987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:37.639529 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97258e3b-22d2-487b-932c-d8dd9432d372","Type":"ContainerStarted","Data":"1063c85644f155ed0d33134bb88f25033f3c875a1e6f5502c0206164c172a0e7"} Nov 01 22:23:37.639987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:37.639557 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97258e3b-22d2-487b-932c-d8dd9432d372","Type":"ContainerStarted","Data":"07262ca6265399476851f9cd64d72995b1d95cf2a312231fc6f4245cd1e7f0cf"} Nov 01 22:23:37.639987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:37.639579 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97258e3b-22d2-487b-932c-d8dd9432d372","Type":"ContainerStarted","Data":"47e7dbbba448a7909548994c527e0e449dee022e829f842418d904ca2d687386"} Nov 01 22:23:37.639987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:37.639601 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97258e3b-22d2-487b-932c-d8dd9432d372","Type":"ContainerStarted","Data":"2efebdbd872a874c0ce5cb01bcab7cfb0be47bf18fcd78c457f6e96ebb0edf91"} Nov 01 22:23:37.639987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:37.639625 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"97258e3b-22d2-487b-932c-d8dd9432d372","Type":"ContainerStarted","Data":"1408cf8dc63a14ee5c14f7dae23cda7214d4a2bb8d578eb47ba7231c31a7dd45"} Nov 01 22:23:37.674145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:37.674016 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.659101697 podStartE2EDuration="2.673998436s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.896739ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:37.672452513 +0000 UTC m=+30.320951761" watchObservedRunningTime="2026-11-01 22:23:37.673998436 +0000 UTC m=+30.322497673" Nov 01 22:23:38.647977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:38.647944 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:38.648402 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:38.648010 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:38.652845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:38.652823 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:39.650443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:39.650416 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:23:39.701552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:39.701522 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc5c8d9fc-q5m4z"] Nov 01 22:23:41.025581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:41.025548 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:23:44.907084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:23:44.907051 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h5fn2" Nov 01 22:23:45.585383 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:23:45.585350 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:24:04.726428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:04.726364 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7cc5c8d9fc-q5m4z" podUID="ad0b4d10-a94f-444b-98b4-465e3618115a" containerName="console" containerID="cri-o://255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2" gracePeriod=15 Nov 01 22:24:04.960546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:04.960524 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:24:05.001944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.001911 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config\") pod \"ad0b4d10-a94f-444b-98b4-465e3618115a\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " Nov 01 22:24:05.002105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.001969 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert\") pod \"ad0b4d10-a94f-444b-98b4-465e3618115a\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " Nov 01 22:24:05.002105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.001993 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm\" (UniqueName: \"kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm\") pod \"ad0b4d10-a94f-444b-98b4-465e3618115a\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " Nov 01 22:24:05.002105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002074 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config\") pod \"ad0b4d10-a94f-444b-98b4-465e3618115a\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " Nov 01 22:24:05.002300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002156 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp\") pod \"ad0b4d10-a94f-444b-98b4-465e3618115a\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " Nov 01 22:24:05.002300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002193 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca\") pod \"ad0b4d10-a94f-444b-98b4-465e3618115a\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " Nov 01 22:24:05.002300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002227 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert\") pod \"ad0b4d10-a94f-444b-98b4-465e3618115a\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " Nov 01 22:24:05.002300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002271 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle\") pod \"ad0b4d10-a94f-444b-98b4-465e3618115a\" (UID: \"ad0b4d10-a94f-444b-98b4-465e3618115a\") " Nov 01 22:24:05.002513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002426 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert" pod "ad0b4d10-a94f-444b-98b4-465e3618115a" (UID: "ad0b4d10-a94f-444b-98b4-465e3618115a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:05.002565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002500 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp" pod "ad0b4d10-a94f-444b-98b4-465e3618115a" (UID: "ad0b4d10-a94f-444b-98b4-465e3618115a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:05.002618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002568 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config" pod "ad0b4d10-a94f-444b-98b4-465e3618115a" (UID: "ad0b4d10-a94f-444b-98b4-465e3618115a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:05.002755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002624 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca" pod "ad0b4d10-a94f-444b-98b4-465e3618115a" (UID: "ad0b4d10-a94f-444b-98b4-465e3618115a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:05.002823 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.002802 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle" pod "ad0b4d10-a94f-444b-98b4-465e3618115a" (UID: "ad0b4d10-a94f-444b-98b4-465e3618115a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:05.004323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.004301 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm" pod "ad0b4d10-a94f-444b-98b4-465e3618115a" (UID: "ad0b4d10-a94f-444b-98b4-465e3618115a"). InnerVolumeSpecName "kube-api-access-ppkjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:05.004323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.004309 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config" pod "ad0b4d10-a94f-444b-98b4-465e3618115a" (UID: "ad0b4d10-a94f-444b-98b4-465e3618115a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:05.004459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.004383 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert" pod "ad0b4d10-a94f-444b-98b4-465e3618115a" (UID: "ad0b4d10-a94f-444b-98b4-465e3618115a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:05.102759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.102736 6287 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-oauth-serving-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:05.102759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.102759 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppkjm\" (UniqueName: \"kubernetes.io/projected/ad0b4d10-a94f-444b-98b4-465e3618115a-kube-api-access-ppkjm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:05.102893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.102771 6287 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-console-config\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:05.102893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.102780 6287 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0b4d10-a94f-444b-98b4-465e3618115a-tmp\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:05.102893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.102789 6287 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:05.102893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.102797 6287 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-serving-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:05.102893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.102806 6287 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0b4d10-a94f-444b-98b4-465e3618115a-trusted-ca-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:05.102893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.102814 6287 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad0b4d10-a94f-444b-98b4-465e3618115a-console-oauth-config\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:05.740199 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.740166 6287 generic.go:356] "Generic (PLEG): container finished" podID="ad0b4d10-a94f-444b-98b4-465e3618115a" containerID="255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2" exitCode=0 Nov 01 22:24:05.740564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.740226 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc5c8d9fc-q5m4z" Nov 01 22:24:05.740564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.740252 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc5c8d9fc-q5m4z" event={"ID":"ad0b4d10-a94f-444b-98b4-465e3618115a","Type":"ContainerDied","Data":"255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2"} Nov 01 22:24:05.740564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.740309 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc5c8d9fc-q5m4z" event={"ID":"ad0b4d10-a94f-444b-98b4-465e3618115a","Type":"ContainerDied","Data":"94e7089b6c49ba7b0e70ce0f1f74fbb7993a42d8bda5427b2e144c6d9dc455f2"} Nov 01 22:24:05.740564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.740330 6287 scope.go:122] "RemoveContainer" containerID="255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2" Nov 01 22:24:05.747968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.747951 6287 scope.go:122] "RemoveContainer" containerID="255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2" Nov 01 22:24:05.748290 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:24:05.748267 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2\": container with ID starting with 255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2 not found: ID does not exist" containerID="255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2" Nov 01 22:24:05.748374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.748295 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2"} err="failed to get container status \"255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2\": rpc error: code = NotFound desc = could not find container \"255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2\": container with ID starting with 255ef9f53c0cc5f2ca29d933407a3701481e567ae441099f2c75d8483101bde2 not found: ID does not exist" Nov 01 22:24:05.755549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.755528 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc5c8d9fc-q5m4z"] Nov 01 22:24:05.759141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:05.759122 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cc5c8d9fc-q5m4z"] Nov 01 22:24:07.466969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:07.466938 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0b4d10-a94f-444b-98b4-465e3618115a" path="/var/lib/kubelet/pods/ad0b4d10-a94f-444b-98b4-465e3618115a/volumes" Nov 01 22:24:15.586149 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:24:15.586100 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:24:34.692162 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:34.692108 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:34.707382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:34.707357 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:34.850422 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:24:34.850394 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:45.585918 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:24:45.585882 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:25:15.585635 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:25:15.585593 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:25:34.544675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.544638 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b"] Nov 01 22:25:34.545232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.545017 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ad0b4d10-a94f-444b-98b4-465e3618115a" containerName="console" Nov 01 22:25:34.545232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.545033 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0b4d10-a94f-444b-98b4-465e3618115a" containerName="console" Nov 01 22:25:34.545232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.545093 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0b4d10-a94f-444b-98b4-465e3618115a" containerName="console" Nov 01 22:25:34.548298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.548276 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.552411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.552385 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 22:25:34.552543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.552523 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 22:25:34.552613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.552573 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-58mdx\"" Nov 01 22:25:34.555763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.555742 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b"] Nov 01 22:25:34.637095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.637070 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.637208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.637186 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxj2\" (UniqueName: \"kubernetes.io/projected/56d72c33-c1de-496e-8707-b6c742c724bc-kube-api-access-cpxj2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.637254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.637222 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.737992 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.737963 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.738091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.737993 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.738091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.738057 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxj2\" (UniqueName: \"kubernetes.io/projected/56d72c33-c1de-496e-8707-b6c742c724bc-kube-api-access-cpxj2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.738319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.738302 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.738386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.738366 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.746522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.746497 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxj2\" (UniqueName: \"kubernetes.io/projected/56d72c33-c1de-496e-8707-b6c742c724bc-kube-api-access-cpxj2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:34.865608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:34.865549 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:25:35.135124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.135030 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b"] Nov 01 22:25:35.138577 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:25:35.138553 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d72c33_c1de_496e_8707_b6c742c724bc.slice/crio-879e3d65da28a5f601be13755d466c27f2d23339c86397ea8002aa49a2b69aad WatchSource:0}: Error finding container 879e3d65da28a5f601be13755d466c27f2d23339c86397ea8002aa49a2b69aad: Status 404 returned error can't find the container with id 879e3d65da28a5f601be13755d466c27f2d23339c86397ea8002aa49a2b69aad Nov 01 22:25:35.140659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.140642 6287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:25:35.248044 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.248017 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp"] Nov 01 22:25:35.255293 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.255249 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.260930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.260907 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp"] Nov 01 22:25:35.261046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.260972 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-8sd4r\"" Nov 01 22:25:35.345577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.345545 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.345761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.345582 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.345761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.345690 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln68f\" (UniqueName: \"kubernetes.io/projected/b74ac2cd-b340-4269-9285-133c9b5c22bd-kube-api-access-ln68f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.348187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.348163 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt"] Nov 01 22:25:35.351378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.351362 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.354351 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.354332 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-vb584\"" Nov 01 22:25:35.359843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.359822 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt"] Nov 01 22:25:35.446756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.446684 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fznh\" (UniqueName: \"kubernetes.io/projected/17b72589-a7e2-45b7-a463-a51979d10858-kube-api-access-8fznh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.446756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.446741 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln68f\" (UniqueName: \"kubernetes.io/projected/b74ac2cd-b340-4269-9285-133c9b5c22bd-kube-api-access-ln68f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.446936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.446821 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.446936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.446866 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.447002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.446931 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.447002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.446967 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.447373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.447351 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.447373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.447356 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.448418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.448399 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686"] Nov 01 22:25:35.451642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.451628 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.454860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.454840 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-6p8r8\"" Nov 01 22:25:35.456280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.456263 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln68f\" (UniqueName: \"kubernetes.io/projected/b74ac2cd-b340-4269-9285-133c9b5c22bd-kube-api-access-ln68f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.459178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.459158 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686"] Nov 01 22:25:35.547817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.547785 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k6t4\" (UniqueName: \"kubernetes.io/projected/5f1260f7-c4b3-415a-ac3a-eb348619ed77-kube-api-access-6k6t4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.548225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.547849 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fznh\" (UniqueName: \"kubernetes.io/projected/17b72589-a7e2-45b7-a463-a51979d10858-kube-api-access-8fznh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.548225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.547898 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.548225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.547953 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.548225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.548041 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.548225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.548071 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.548483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.548406 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.548483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.548462 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.554380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.554360 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk"] Nov 01 22:25:35.558202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.558188 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.561771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.561747 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fznh\" (UniqueName: \"kubernetes.io/projected/17b72589-a7e2-45b7-a463-a51979d10858-kube-api-access-8fznh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.566267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.566234 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk"] Nov 01 22:25:35.566410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.566396 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-wsl9s\"" Nov 01 22:25:35.573277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.573258 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:25:35.648728 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.648687 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6t4\" (UniqueName: \"kubernetes.io/projected/5f1260f7-c4b3-415a-ac3a-eb348619ed77-kube-api-access-6k6t4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.648878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.648741 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.648878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.648772 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.648878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.648804 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pv6p\" (UniqueName: \"kubernetes.io/projected/ccbe653a-5cd2-4a58-a4fd-20a905296842-kube-api-access-8pv6p\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.648878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.648824 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.649083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.648966 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.649159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.649095 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.649267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.649250 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.656551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.656531 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6t4\" (UniqueName: \"kubernetes.io/projected/5f1260f7-c4b3-415a-ac3a-eb348619ed77-kube-api-access-6k6t4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.667292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.667272 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:25:35.751144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.749747 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pv6p\" (UniqueName: \"kubernetes.io/projected/ccbe653a-5cd2-4a58-a4fd-20a905296842-kube-api-access-8pv6p\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.751144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.749791 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.751144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.749880 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.751144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.750315 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.751144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.750364 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.757835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.757814 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pv6p\" (UniqueName: \"kubernetes.io/projected/ccbe653a-5cd2-4a58-a4fd-20a905296842-kube-api-access-8pv6p\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.774546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.774526 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:25:35.839707 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.839679 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp"] Nov 01 22:25:35.846213 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:25:35.844084 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74ac2cd_b340_4269_9285_133c9b5c22bd.slice/crio-1fe67e4f66c124e9d6f327b5cce1f29d9a12fbf3dfc9dbe5a55e940ef7200fde WatchSource:0}: Error finding container 1fe67e4f66c124e9d6f327b5cce1f29d9a12fbf3dfc9dbe5a55e940ef7200fde: Status 404 returned error can't find the container with id 1fe67e4f66c124e9d6f327b5cce1f29d9a12fbf3dfc9dbe5a55e940ef7200fde Nov 01 22:25:35.874703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.874680 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:25:35.941048 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:35.941021 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt"] Nov 01 22:25:35.945205 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:25:35.945163 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b72589_a7e2_45b7_a463_a51979d10858.slice/crio-81e62cfa4373316b7d5489c42d7f098df4a7bf81d587135e503af660f5f1c2ff WatchSource:0}: Error finding container 81e62cfa4373316b7d5489c42d7f098df4a7bf81d587135e503af660f5f1c2ff: Status 404 returned error can't find the container with id 81e62cfa4373316b7d5489c42d7f098df4a7bf81d587135e503af660f5f1c2ff Nov 01 22:25:36.027379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:36.027349 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" event={"ID":"17b72589-a7e2-45b7-a463-a51979d10858","Type":"ContainerStarted","Data":"81e62cfa4373316b7d5489c42d7f098df4a7bf81d587135e503af660f5f1c2ff"} Nov 01 22:25:36.028589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:36.028559 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" event={"ID":"b74ac2cd-b340-4269-9285-133c9b5c22bd","Type":"ContainerStarted","Data":"1fe67e4f66c124e9d6f327b5cce1f29d9a12fbf3dfc9dbe5a55e940ef7200fde"} Nov 01 22:25:36.029785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:36.029762 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" event={"ID":"56d72c33-c1de-496e-8707-b6c742c724bc","Type":"ContainerStarted","Data":"879e3d65da28a5f601be13755d466c27f2d23339c86397ea8002aa49a2b69aad"} Nov 01 22:25:36.043542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:36.043514 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686"] Nov 01 22:25:36.046469 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:25:36.046447 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1260f7_c4b3_415a_ac3a_eb348619ed77.slice/crio-2b746fdf9832475cbbb6472498b3de9de612ba83abdf124a0a9b0b7fb9cce08e WatchSource:0}: Error finding container 2b746fdf9832475cbbb6472498b3de9de612ba83abdf124a0a9b0b7fb9cce08e: Status 404 returned error can't find the container with id 2b746fdf9832475cbbb6472498b3de9de612ba83abdf124a0a9b0b7fb9cce08e Nov 01 22:25:36.144745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:36.144720 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk"] Nov 01 22:25:36.147833 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:25:36.147809 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbe653a_5cd2_4a58_a4fd_20a905296842.slice/crio-b5a89b3fad31b114b3a8f14ae92d8c764d84ceeb33302bed0345cc1546e4b0e2 WatchSource:0}: Error finding container b5a89b3fad31b114b3a8f14ae92d8c764d84ceeb33302bed0345cc1546e4b0e2: Status 404 returned error can't find the container with id b5a89b3fad31b114b3a8f14ae92d8c764d84ceeb33302bed0345cc1546e4b0e2 Nov 01 22:25:37.037661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.037625 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" event={"ID":"ccbe653a-5cd2-4a58-a4fd-20a905296842","Type":"ContainerStarted","Data":"b5a89b3fad31b114b3a8f14ae92d8c764d84ceeb33302bed0345cc1546e4b0e2"} Nov 01 22:25:37.039275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.039246 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" event={"ID":"5f1260f7-c4b3-415a-ac3a-eb348619ed77","Type":"ContainerStarted","Data":"2b746fdf9832475cbbb6472498b3de9de612ba83abdf124a0a9b0b7fb9cce08e"} Nov 01 22:25:37.148232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.148182 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds"] Nov 01 22:25:37.151751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.151725 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.161712 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.161659 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 22:25:37.161854 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.161813 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-flhqn\"" Nov 01 22:25:37.162259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.162235 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 22:25:37.162344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.162328 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 22:25:37.167929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.167213 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds"] Nov 01 22:25:37.269734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.269704 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.269901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.269748 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc7pb\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-kube-api-access-kc7pb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.269901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.269875 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.270011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.269910 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.270011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.269962 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.270011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.269998 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71935876-95e7-4691-bedf-e040f4cfec12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.270188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.270026 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71935876-95e7-4691-bedf-e040f4cfec12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.270188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.270056 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.270188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.270151 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.371288 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.371192 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.371288 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.371250 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.371492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.371304 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.371492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.371343 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71935876-95e7-4691-bedf-e040f4cfec12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.371492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.371368 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71935876-95e7-4691-bedf-e040f4cfec12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.371492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.371396 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.371492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.371426 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.371492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.371480 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.371741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.371505 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc7pb\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-kube-api-access-kc7pb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.372306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.372278 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.372556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.372536 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.372830 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.372778 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.374369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.373427 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.374369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.374043 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71935876-95e7-4691-bedf-e040f4cfec12-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.382367 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.382345 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71935876-95e7-4691-bedf-e040f4cfec12-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.382705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.382620 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.388375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.388353 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.392210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.392186 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc7pb\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-kube-api-access-kc7pb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-87hds\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.473000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.472790 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:37.576481 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.576411 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws"] Nov 01 22:25:37.607905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.604342 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws"] Nov 01 22:25:37.607905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.604493 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.609247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.609222 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-4n47r\"" Nov 01 22:25:37.675942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.675836 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwrrw\" (UniqueName: \"kubernetes.io/projected/b7f5a86b-b367-4d7e-986a-c27bd976ca18-kube-api-access-nwrrw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.675942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.675889 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.675942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.675932 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.780407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.778735 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwrrw\" (UniqueName: \"kubernetes.io/projected/b7f5a86b-b367-4d7e-986a-c27bd976ca18-kube-api-access-nwrrw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.780407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.778868 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.780407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.778915 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.780407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.779561 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.780407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.780355 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.797100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.797071 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwrrw\" (UniqueName: \"kubernetes.io/projected/b7f5a86b-b367-4d7e-986a-c27bd976ca18-kube-api-access-nwrrw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:37.814323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.814295 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds"] Nov 01 22:25:37.819719 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:25:37.819657 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71935876_95e7_4691_bedf_e040f4cfec12.slice/crio-e25e0fc34a38e8a8d7546d4a4b592f75731291638e10dea5dd255f437d599434 WatchSource:0}: Error finding container e25e0fc34a38e8a8d7546d4a4b592f75731291638e10dea5dd255f437d599434: Status 404 returned error can't find the container with id e25e0fc34a38e8a8d7546d4a4b592f75731291638e10dea5dd255f437d599434 Nov 01 22:25:37.938370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:37.938299 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:25:38.053456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.053415 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" event={"ID":"71935876-95e7-4691-bedf-e040f4cfec12","Type":"ContainerStarted","Data":"e25e0fc34a38e8a8d7546d4a4b592f75731291638e10dea5dd255f437d599434"} Nov 01 22:25:38.228300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.228221 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws"] Nov 01 22:25:38.237379 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:25:38.237308 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f5a86b_b367_4d7e_986a_c27bd976ca18.slice/crio-eb724b8211f1a827c9f524c4aaabb527c9f5d78100695703a4c205c4363ead2c WatchSource:0}: Error finding container eb724b8211f1a827c9f524c4aaabb527c9f5d78100695703a4c205c4363ead2c: Status 404 returned error can't find the container with id eb724b8211f1a827c9f524c4aaabb527c9f5d78100695703a4c205c4363ead2c Nov 01 22:25:38.430210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.430176 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw"] Nov 01 22:25:38.435515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.435491 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.440771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.440688 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw"] Nov 01 22:25:38.441367 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.441312 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-5kx79\"" Nov 01 22:25:38.584808 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.584775 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.584996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.584812 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.584996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.584865 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcbw\" (UniqueName: \"kubernetes.io/projected/19a5da99-824a-4f97-a26c-c6507e083cff-kube-api-access-bbcbw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.686039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.685990 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcbw\" (UniqueName: \"kubernetes.io/projected/19a5da99-824a-4f97-a26c-c6507e083cff-kube-api-access-bbcbw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.686217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.686082 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.686217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.686122 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.686493 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.686469 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.687059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.687032 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.701289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.701262 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcbw\" (UniqueName: \"kubernetes.io/projected/19a5da99-824a-4f97-a26c-c6507e083cff-kube-api-access-bbcbw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:38.761953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:38.761924 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:25:39.068057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:39.067962 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" event={"ID":"b7f5a86b-b367-4d7e-986a-c27bd976ca18","Type":"ContainerStarted","Data":"eb724b8211f1a827c9f524c4aaabb527c9f5d78100695703a4c205c4363ead2c"} Nov 01 22:25:39.073128 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:39.072221 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw"] Nov 01 22:25:39.099918 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:25:39.099867 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a5da99_824a_4f97_a26c_c6507e083cff.slice/crio-534396c6046902423e40c6ecfe3124dc9178524e30af4bffbf22985e62ae8cb4 WatchSource:0}: Error finding container 534396c6046902423e40c6ecfe3124dc9178524e30af4bffbf22985e62ae8cb4: Status 404 returned error can't find the container with id 534396c6046902423e40c6ecfe3124dc9178524e30af4bffbf22985e62ae8cb4 Nov 01 22:25:40.085567 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:40.085491 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" event={"ID":"19a5da99-824a-4f97-a26c-c6507e083cff","Type":"ContainerStarted","Data":"534396c6046902423e40c6ecfe3124dc9178524e30af4bffbf22985e62ae8cb4"} Nov 01 22:25:45.567828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:45.563979 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:25:45.567828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:45.564066 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:25:45.567828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:45.564125 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:25:45.615178 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:25:45.615143 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:25:46.138652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:46.138615 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" event={"ID":"71935876-95e7-4691-bedf-e040f4cfec12","Type":"ContainerStarted","Data":"2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c"} Nov 01 22:25:46.474058 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:46.473975 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:46.480238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:46.480180 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:46.500903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:46.500035 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" podStartSLOduration=1.75970034 podStartE2EDuration="9.500020044s" totalImagesPullingTime="7.740319704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:46.164382794 +0000 UTC m=+158.812882030" watchObservedRunningTime="2026-11-01 22:25:46.500020044 +0000 UTC m=+159.148519279" Nov 01 22:25:47.145678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:47.145640 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:47.146625 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:47.146600 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:25:50.166577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:50.166503 6287 generic.go:356] "Generic (PLEG): container finished" podID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerID="daafe5606a49b50a7887547516960116dcc7feed14e2dbf6e355caf7e85b9a7b" exitCode=0 Nov 01 22:25:50.167091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:50.166634 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" event={"ID":"b74ac2cd-b340-4269-9285-133c9b5c22bd","Type":"ContainerDied","Data":"daafe5606a49b50a7887547516960116dcc7feed14e2dbf6e355caf7e85b9a7b"} Nov 01 22:25:50.170069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:50.169373 6287 generic.go:356] "Generic (PLEG): container finished" podID="17b72589-a7e2-45b7-a463-a51979d10858" containerID="370747a979655e1ed72bdd4ebe621fc829853be68d2e348af5385faa38d17f9e" exitCode=0 Nov 01 22:25:50.170069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:50.169412 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" event={"ID":"17b72589-a7e2-45b7-a463-a51979d10858","Type":"ContainerDied","Data":"370747a979655e1ed72bdd4ebe621fc829853be68d2e348af5385faa38d17f9e"} Nov 01 22:25:51.175906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:51.175807 6287 generic.go:356] "Generic (PLEG): container finished" podID="56d72c33-c1de-496e-8707-b6c742c724bc" containerID="52b7a80af1f90b593c9a97e214405c07695f83a2f2b63f42adbcb92bed57653f" exitCode=0 Nov 01 22:25:51.176383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:51.175977 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" event={"ID":"56d72c33-c1de-496e-8707-b6c742c724bc","Type":"ContainerDied","Data":"52b7a80af1f90b593c9a97e214405c07695f83a2f2b63f42adbcb92bed57653f"} Nov 01 22:25:51.178821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:51.178776 6287 generic.go:356] "Generic (PLEG): container finished" podID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerID="215e215d9fbee7a64f1e0b8166ed28acba64555fc1eb9c4d54fbc8b19b31ba87" exitCode=0 Nov 01 22:25:51.178916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:51.178877 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" event={"ID":"ccbe653a-5cd2-4a58-a4fd-20a905296842","Type":"ContainerDied","Data":"215e215d9fbee7a64f1e0b8166ed28acba64555fc1eb9c4d54fbc8b19b31ba87"} Nov 01 22:25:51.181890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:51.181861 6287 generic.go:356] "Generic (PLEG): container finished" podID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerID="6c78a100387a9ef723096a12375481f12dc46f0e44bf721ca2855d18a3f1db5c" exitCode=0 Nov 01 22:25:51.181996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:51.181905 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" event={"ID":"5f1260f7-c4b3-415a-ac3a-eb348619ed77","Type":"ContainerDied","Data":"6c78a100387a9ef723096a12375481f12dc46f0e44bf721ca2855d18a3f1db5c"} Nov 01 22:25:52.186927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:52.186900 6287 generic.go:356] "Generic (PLEG): container finished" podID="19a5da99-824a-4f97-a26c-c6507e083cff" containerID="08ff1aec2228d63a82a32b9a2c44d9ea7c6765ee0b25e603e9c0881784b4f217" exitCode=0 Nov 01 22:25:52.187343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:52.186985 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" event={"ID":"19a5da99-824a-4f97-a26c-c6507e083cff","Type":"ContainerDied","Data":"08ff1aec2228d63a82a32b9a2c44d9ea7c6765ee0b25e603e9c0881784b4f217"} Nov 01 22:25:52.188905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:52.188845 6287 generic.go:356] "Generic (PLEG): container finished" podID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerID="04b0273b4291d1417586e568382303639fb0ae01ad97931f16ae5c23a64c63e7" exitCode=0 Nov 01 22:25:52.188985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:52.188906 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" event={"ID":"b74ac2cd-b340-4269-9285-133c9b5c22bd","Type":"ContainerDied","Data":"04b0273b4291d1417586e568382303639fb0ae01ad97931f16ae5c23a64c63e7"} Nov 01 22:25:52.191263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:52.191243 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" event={"ID":"b7f5a86b-b367-4d7e-986a-c27bd976ca18","Type":"ContainerStarted","Data":"f4e7398f22e8e1f8d9018c0eeb147c024a9a47cb6e43b269464e25ec738bf4e1"} Nov 01 22:25:52.193097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:52.193074 6287 generic.go:356] "Generic (PLEG): container finished" podID="17b72589-a7e2-45b7-a463-a51979d10858" containerID="860dfc3a226910df716083209bf41e3dc2a492f4641f37e0573dcad21a6eb09d" exitCode=0 Nov 01 22:25:52.193204 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:52.193132 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" event={"ID":"17b72589-a7e2-45b7-a463-a51979d10858","Type":"ContainerDied","Data":"860dfc3a226910df716083209bf41e3dc2a492f4641f37e0573dcad21a6eb09d"} Nov 01 22:25:53.206073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:53.206032 6287 generic.go:356] "Generic (PLEG): container finished" podID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerID="f4e7398f22e8e1f8d9018c0eeb147c024a9a47cb6e43b269464e25ec738bf4e1" exitCode=0 Nov 01 22:25:53.206638 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:53.206175 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" event={"ID":"b7f5a86b-b367-4d7e-986a-c27bd976ca18","Type":"ContainerDied","Data":"f4e7398f22e8e1f8d9018c0eeb147c024a9a47cb6e43b269464e25ec738bf4e1"} Nov 01 22:25:53.209835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:53.208859 6287 generic.go:356] "Generic (PLEG): container finished" podID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerID="49634bd9d5ce84278ddefb353a7ad287b33aae41fb0e7bc7f8ff4d0b4e2a4a6d" exitCode=0 Nov 01 22:25:53.209835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:53.208942 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" event={"ID":"ccbe653a-5cd2-4a58-a4fd-20a905296842","Type":"ContainerDied","Data":"49634bd9d5ce84278ddefb353a7ad287b33aae41fb0e7bc7f8ff4d0b4e2a4a6d"} Nov 01 22:25:53.213348 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:53.213322 6287 generic.go:356] "Generic (PLEG): container finished" podID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerID="46c5ba4da57415ca3674b5d02428428062e4b931110127450b37278457ba9f69" exitCode=0 Nov 01 22:25:53.213440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:53.213405 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" event={"ID":"5f1260f7-c4b3-415a-ac3a-eb348619ed77","Type":"ContainerDied","Data":"46c5ba4da57415ca3674b5d02428428062e4b931110127450b37278457ba9f69"} Nov 01 22:25:53.217225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:53.217201 6287 generic.go:356] "Generic (PLEG): container finished" podID="19a5da99-824a-4f97-a26c-c6507e083cff" containerID="4ce44d65741559f8b0ea16bac32a504901d6d64b173b0de87d465d6d69375c47" exitCode=0 Nov 01 22:25:53.217319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:53.217245 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" event={"ID":"19a5da99-824a-4f97-a26c-c6507e083cff","Type":"ContainerDied","Data":"4ce44d65741559f8b0ea16bac32a504901d6d64b173b0de87d465d6d69375c47"} Nov 01 22:25:57.266175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:57.265558 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" event={"ID":"56d72c33-c1de-496e-8707-b6c742c724bc","Type":"ContainerStarted","Data":"e172b5d0b90c154097942943e26d33507d67f9a208a01a25d710a530530fa92c"} Nov 01 22:25:57.273089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:57.273012 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" event={"ID":"b7f5a86b-b367-4d7e-986a-c27bd976ca18","Type":"ContainerStarted","Data":"af047b952a3df084501aa00d860dd95a47da0312d073a25eb8c3adcbb0f8e0f0"} Nov 01 22:25:58.285552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:58.285511 6287 generic.go:356] "Generic (PLEG): container finished" podID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerID="af047b952a3df084501aa00d860dd95a47da0312d073a25eb8c3adcbb0f8e0f0" exitCode=0 Nov 01 22:25:58.287954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:58.287924 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" event={"ID":"b7f5a86b-b367-4d7e-986a-c27bd976ca18","Type":"ContainerDied","Data":"af047b952a3df084501aa00d860dd95a47da0312d073a25eb8c3adcbb0f8e0f0"} Nov 01 22:25:58.293220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:58.293188 6287 generic.go:356] "Generic (PLEG): container finished" podID="56d72c33-c1de-496e-8707-b6c742c724bc" containerID="e172b5d0b90c154097942943e26d33507d67f9a208a01a25d710a530530fa92c" exitCode=0 Nov 01 22:25:58.293327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:25:58.293236 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" event={"ID":"56d72c33-c1de-496e-8707-b6c742c724bc","Type":"ContainerDied","Data":"e172b5d0b90c154097942943e26d33507d67f9a208a01a25d710a530530fa92c"} Nov 01 22:26:06.353165 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:06.353072 6287 generic.go:356] "Generic (PLEG): container finished" podID="17b72589-a7e2-45b7-a463-a51979d10858" containerID="6f725e46caf7ebbd5efc13fde9e1fb678302c18964e82f23523c1479e05cb47a" exitCode=0 Nov 01 22:26:06.353650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:06.353187 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" event={"ID":"17b72589-a7e2-45b7-a463-a51979d10858","Type":"ContainerDied","Data":"6f725e46caf7ebbd5efc13fde9e1fb678302c18964e82f23523c1479e05cb47a"} Nov 01 22:26:07.504883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.504854 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:26:07.569345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.569314 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b72589-a7e2-45b7-a463-a51979d10858-kube-api-access-8fznh\" (UniqueName: \"kubernetes.io/projected/17b72589-a7e2-45b7-a463-a51979d10858-kube-api-access-8fznh\") pod \"17b72589-a7e2-45b7-a463-a51979d10858\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " Nov 01 22:26:07.569502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.569352 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-bundle\") pod \"17b72589-a7e2-45b7-a463-a51979d10858\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " Nov 01 22:26:07.569502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.569436 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-util\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-util\") pod \"17b72589-a7e2-45b7-a463-a51979d10858\" (UID: \"17b72589-a7e2-45b7-a463-a51979d10858\") " Nov 01 22:26:07.569860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.569833 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-bundle" pod "17b72589-a7e2-45b7-a463-a51979d10858" (UID: "17b72589-a7e2-45b7-a463-a51979d10858"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:07.571504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.571473 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b72589-a7e2-45b7-a463-a51979d10858-kube-api-access-8fznh" pod "17b72589-a7e2-45b7-a463-a51979d10858" (UID: "17b72589-a7e2-45b7-a463-a51979d10858"). InnerVolumeSpecName "kube-api-access-8fznh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:07.573446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.573426 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-util" pod "17b72589-a7e2-45b7-a463-a51979d10858" (UID: "17b72589-a7e2-45b7-a463-a51979d10858"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:07.670535 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.670505 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fznh\" (UniqueName: \"kubernetes.io/projected/17b72589-a7e2-45b7-a463-a51979d10858-kube-api-access-8fznh\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:07.670642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.670537 6287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:07.670642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:07.670554 6287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17b72589-a7e2-45b7-a463-a51979d10858-util\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:08.363480 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.363436 6287 generic.go:356] "Generic (PLEG): container finished" podID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerID="05ac8689f0b2bf9eb88743e65e46b304368a0afeb3d06f2e1ed356d2d1ebc242" exitCode=0 Nov 01 22:26:08.363680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.363525 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" event={"ID":"b74ac2cd-b340-4269-9285-133c9b5c22bd","Type":"ContainerDied","Data":"05ac8689f0b2bf9eb88743e65e46b304368a0afeb3d06f2e1ed356d2d1ebc242"} Nov 01 22:26:08.365482 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.365459 6287 generic.go:356] "Generic (PLEG): container finished" podID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerID="02a2fbb47a9dd177f219533c132f2b30c2f87b877203b1bd0547ccb46454e6f6" exitCode=0 Nov 01 22:26:08.365620 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.365533 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" event={"ID":"b7f5a86b-b367-4d7e-986a-c27bd976ca18","Type":"ContainerDied","Data":"02a2fbb47a9dd177f219533c132f2b30c2f87b877203b1bd0547ccb46454e6f6"} Nov 01 22:26:08.367528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.367502 6287 generic.go:356] "Generic (PLEG): container finished" podID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerID="915c0b21626216f88b767ccfd57c23fb2f3284b8a1273e76d337687f3886f8d8" exitCode=0 Nov 01 22:26:08.367658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.367574 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" event={"ID":"ccbe653a-5cd2-4a58-a4fd-20a905296842","Type":"ContainerDied","Data":"915c0b21626216f88b767ccfd57c23fb2f3284b8a1273e76d337687f3886f8d8"} Nov 01 22:26:08.369541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.369518 6287 generic.go:356] "Generic (PLEG): container finished" podID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerID="6be890591a6b77c4036a453834f4e366ef4aa92c82edba058fe4ac9027aa8310" exitCode=0 Nov 01 22:26:08.369661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.369578 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" event={"ID":"5f1260f7-c4b3-415a-ac3a-eb348619ed77","Type":"ContainerDied","Data":"6be890591a6b77c4036a453834f4e366ef4aa92c82edba058fe4ac9027aa8310"} Nov 01 22:26:08.371312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.371291 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" event={"ID":"17b72589-a7e2-45b7-a463-a51979d10858","Type":"ContainerDied","Data":"81e62cfa4373316b7d5489c42d7f098df4a7bf81d587135e503af660f5f1c2ff"} Nov 01 22:26:08.371424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.371327 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81e62cfa4373316b7d5489c42d7f098df4a7bf81d587135e503af660f5f1c2ff" Nov 01 22:26:08.371424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.371304 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mhpdt" Nov 01 22:26:08.373299 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.373278 6287 generic.go:356] "Generic (PLEG): container finished" podID="56d72c33-c1de-496e-8707-b6c742c724bc" containerID="28bda08a48e892f239fa1d78a11360f1526be55ed151a8cbcc26e913a54daf3d" exitCode=0 Nov 01 22:26:08.373393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.373354 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" event={"ID":"56d72c33-c1de-496e-8707-b6c742c724bc","Type":"ContainerDied","Data":"28bda08a48e892f239fa1d78a11360f1526be55ed151a8cbcc26e913a54daf3d"} Nov 01 22:26:08.375443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.375423 6287 generic.go:356] "Generic (PLEG): container finished" podID="19a5da99-824a-4f97-a26c-c6507e083cff" containerID="a145172e7ecd82e69f743fd706be633e3469999fbd7ca75663fbe4e7dd5dea79" exitCode=0 Nov 01 22:26:08.375521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:08.375467 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" event={"ID":"19a5da99-824a-4f97-a26c-c6507e083cff","Type":"ContainerDied","Data":"a145172e7ecd82e69f743fd706be633e3469999fbd7ca75663fbe4e7dd5dea79"} Nov 01 22:26:09.534659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.534634 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:26:09.587417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.586886 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-util\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-util\") pod \"19a5da99-824a-4f97-a26c-c6507e083cff\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " Nov 01 22:26:09.587417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.586956 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a5da99-824a-4f97-a26c-c6507e083cff-kube-api-access-bbcbw\" (UniqueName: \"kubernetes.io/projected/19a5da99-824a-4f97-a26c-c6507e083cff-kube-api-access-bbcbw\") pod \"19a5da99-824a-4f97-a26c-c6507e083cff\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " Nov 01 22:26:09.587417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.586981 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-bundle\") pod \"19a5da99-824a-4f97-a26c-c6507e083cff\" (UID: \"19a5da99-824a-4f97-a26c-c6507e083cff\") " Nov 01 22:26:09.589508 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.588887 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-bundle" pod "19a5da99-824a-4f97-a26c-c6507e083cff" (UID: "19a5da99-824a-4f97-a26c-c6507e083cff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.592792 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.592768 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-util" pod "19a5da99-824a-4f97-a26c-c6507e083cff" (UID: "19a5da99-824a-4f97-a26c-c6507e083cff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.594720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.594540 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:26:09.594720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.594674 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a5da99-824a-4f97-a26c-c6507e083cff-kube-api-access-bbcbw" pod "19a5da99-824a-4f97-a26c-c6507e083cff" (UID: "19a5da99-824a-4f97-a26c-c6507e083cff"). InnerVolumeSpecName "kube-api-access-bbcbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:09.609832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.609792 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:26:09.616236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.615378 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:26:09.621286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.621263 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:26:09.622243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.622230 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:26:09.688654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688574 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-bundle\") pod \"56d72c33-c1de-496e-8707-b6c742c724bc\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " Nov 01 22:26:09.688654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688618 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1260f7-c4b3-415a-ac3a-eb348619ed77-kube-api-access-6k6t4\" (UniqueName: \"kubernetes.io/projected/5f1260f7-c4b3-415a-ac3a-eb348619ed77-kube-api-access-6k6t4\") pod \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " Nov 01 22:26:09.688654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688636 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbe653a-5cd2-4a58-a4fd-20a905296842-kube-api-access-8pv6p\" (UniqueName: \"kubernetes.io/projected/ccbe653a-5cd2-4a58-a4fd-20a905296842-kube-api-access-8pv6p\") pod \"ccbe653a-5cd2-4a58-a4fd-20a905296842\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " Nov 01 22:26:09.688936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688668 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74ac2cd-b340-4269-9285-133c9b5c22bd-kube-api-access-ln68f\" (UniqueName: \"kubernetes.io/projected/b74ac2cd-b340-4269-9285-133c9b5c22bd-kube-api-access-ln68f\") pod \"b74ac2cd-b340-4269-9285-133c9b5c22bd\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " Nov 01 22:26:09.688936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688701 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-bundle\") pod \"ccbe653a-5cd2-4a58-a4fd-20a905296842\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " Nov 01 22:26:09.688936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688731 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-bundle\") pod \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " Nov 01 22:26:09.688936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688823 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f5a86b-b367-4d7e-986a-c27bd976ca18-kube-api-access-nwrrw\" (UniqueName: \"kubernetes.io/projected/b7f5a86b-b367-4d7e-986a-c27bd976ca18-kube-api-access-nwrrw\") pod \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " Nov 01 22:26:09.688936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688918 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d72c33-c1de-496e-8707-b6c742c724bc-kube-api-access-cpxj2\" (UniqueName: \"kubernetes.io/projected/56d72c33-c1de-496e-8707-b6c742c724bc-kube-api-access-cpxj2\") pod \"56d72c33-c1de-496e-8707-b6c742c724bc\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " Nov 01 22:26:09.689218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688957 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-bundle\") pod \"b74ac2cd-b340-4269-9285-133c9b5c22bd\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " Nov 01 22:26:09.689218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.688991 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-util\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-util\") pod \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\" (UID: \"5f1260f7-c4b3-415a-ac3a-eb348619ed77\") " Nov 01 22:26:09.689218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689036 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-util\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-util\") pod \"56d72c33-c1de-496e-8707-b6c742c724bc\" (UID: \"56d72c33-c1de-496e-8707-b6c742c724bc\") " Nov 01 22:26:09.689218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689066 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-util\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-util\") pod \"b74ac2cd-b340-4269-9285-133c9b5c22bd\" (UID: \"b74ac2cd-b340-4269-9285-133c9b5c22bd\") " Nov 01 22:26:09.689218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689133 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-util\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-util\") pod \"ccbe653a-5cd2-4a58-a4fd-20a905296842\" (UID: \"ccbe653a-5cd2-4a58-a4fd-20a905296842\") " Nov 01 22:26:09.689218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689149 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-bundle" pod "56d72c33-c1de-496e-8707-b6c742c724bc" (UID: "56d72c33-c1de-496e-8707-b6c742c724bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.689218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689160 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-util\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-util\") pod \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " Nov 01 22:26:09.689568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689228 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-bundle" pod "5f1260f7-c4b3-415a-ac3a-eb348619ed77" (UID: "5f1260f7-c4b3-415a-ac3a-eb348619ed77"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.689568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689243 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-bundle\") pod \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\" (UID: \"b7f5a86b-b367-4d7e-986a-c27bd976ca18\") " Nov 01 22:26:09.689672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689612 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-bundle" pod "ccbe653a-5cd2-4a58-a4fd-20a905296842" (UID: "ccbe653a-5cd2-4a58-a4fd-20a905296842"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.689672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689619 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbcbw\" (UniqueName: \"kubernetes.io/projected/19a5da99-824a-4f97-a26c-c6507e083cff-kube-api-access-bbcbw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.689672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689644 6287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.689672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689659 6287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.689903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689674 6287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.689903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.689687 6287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a5da99-824a-4f97-a26c-c6507e083cff-util\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.690763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.690726 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-bundle" pod "b74ac2cd-b340-4269-9285-133c9b5c22bd" (UID: "b74ac2cd-b340-4269-9285-133c9b5c22bd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.691645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.691617 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-bundle" pod "b7f5a86b-b367-4d7e-986a-c27bd976ca18" (UID: "b7f5a86b-b367-4d7e-986a-c27bd976ca18"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.691926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.691903 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74ac2cd-b340-4269-9285-133c9b5c22bd-kube-api-access-ln68f" pod "b74ac2cd-b340-4269-9285-133c9b5c22bd" (UID: "b74ac2cd-b340-4269-9285-133c9b5c22bd"). InnerVolumeSpecName "kube-api-access-ln68f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:09.692049 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.692020 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d72c33-c1de-496e-8707-b6c742c724bc-kube-api-access-cpxj2" pod "56d72c33-c1de-496e-8707-b6c742c724bc" (UID: "56d72c33-c1de-496e-8707-b6c742c724bc"). InnerVolumeSpecName "kube-api-access-cpxj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:09.693409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.693384 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbe653a-5cd2-4a58-a4fd-20a905296842-kube-api-access-8pv6p" pod "ccbe653a-5cd2-4a58-a4fd-20a905296842" (UID: "ccbe653a-5cd2-4a58-a4fd-20a905296842"). InnerVolumeSpecName "kube-api-access-8pv6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:09.693631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.693608 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1260f7-c4b3-415a-ac3a-eb348619ed77-kube-api-access-6k6t4" pod "5f1260f7-c4b3-415a-ac3a-eb348619ed77" (UID: "5f1260f7-c4b3-415a-ac3a-eb348619ed77"). InnerVolumeSpecName "kube-api-access-6k6t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:09.693824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.693800 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f5a86b-b367-4d7e-986a-c27bd976ca18-kube-api-access-nwrrw" pod "b7f5a86b-b367-4d7e-986a-c27bd976ca18" (UID: "b7f5a86b-b367-4d7e-986a-c27bd976ca18"). InnerVolumeSpecName "kube-api-access-nwrrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:26:09.695680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.695658 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-util" pod "5f1260f7-c4b3-415a-ac3a-eb348619ed77" (UID: "5f1260f7-c4b3-415a-ac3a-eb348619ed77"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.697570 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.697551 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-util" pod "ccbe653a-5cd2-4a58-a4fd-20a905296842" (UID: "ccbe653a-5cd2-4a58-a4fd-20a905296842"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.697617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.697570 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-util" pod "56d72c33-c1de-496e-8707-b6c742c724bc" (UID: "56d72c33-c1de-496e-8707-b6c742c724bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.697994 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.697972 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-util" pod "b7f5a86b-b367-4d7e-986a-c27bd976ca18" (UID: "b7f5a86b-b367-4d7e-986a-c27bd976ca18"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.698681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.698663 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-util" pod "b74ac2cd-b340-4269-9285-133c9b5c22bd" (UID: "b74ac2cd-b340-4269-9285-133c9b5c22bd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:26:09.790982 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.790942 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxj2\" (UniqueName: \"kubernetes.io/projected/56d72c33-c1de-496e-8707-b6c742c724bc-kube-api-access-cpxj2\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.790982 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.790976 6287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.790982 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.790989 6287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f1260f7-c4b3-415a-ac3a-eb348619ed77-util\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791001 6287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56d72c33-c1de-496e-8707-b6c742c724bc-util\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791011 6287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b74ac2cd-b340-4269-9285-133c9b5c22bd-util\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791022 6287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-util\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791033 6287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-util\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791043 6287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f5a86b-b367-4d7e-986a-c27bd976ca18-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791055 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k6t4\" (UniqueName: \"kubernetes.io/projected/5f1260f7-c4b3-415a-ac3a-eb348619ed77-kube-api-access-6k6t4\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791068 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pv6p\" (UniqueName: \"kubernetes.io/projected/ccbe653a-5cd2-4a58-a4fd-20a905296842-kube-api-access-8pv6p\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791080 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln68f\" (UniqueName: \"kubernetes.io/projected/b74ac2cd-b340-4269-9285-133c9b5c22bd-kube-api-access-ln68f\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791091 6287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccbe653a-5cd2-4a58-a4fd-20a905296842-bundle\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:09.791255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:09.791104 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwrrw\" (UniqueName: \"kubernetes.io/projected/b7f5a86b-b367-4d7e-986a-c27bd976ca18-kube-api-access-nwrrw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:26:10.384596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.384554 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" event={"ID":"b74ac2cd-b340-4269-9285-133c9b5c22bd","Type":"ContainerDied","Data":"1fe67e4f66c124e9d6f327b5cce1f29d9a12fbf3dfc9dbe5a55e940ef7200fde"} Nov 01 22:26:10.384773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.384607 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fe67e4f66c124e9d6f327b5cce1f29d9a12fbf3dfc9dbe5a55e940ef7200fde" Nov 01 22:26:10.384773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.384570 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf8bp" Nov 01 22:26:10.386577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.386559 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" Nov 01 22:26:10.386703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.386577 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2clws" event={"ID":"b7f5a86b-b367-4d7e-986a-c27bd976ca18","Type":"ContainerDied","Data":"eb724b8211f1a827c9f524c4aaabb527c9f5d78100695703a4c205c4363ead2c"} Nov 01 22:26:10.386703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.386616 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb724b8211f1a827c9f524c4aaabb527c9f5d78100695703a4c205c4363ead2c" Nov 01 22:26:10.388611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.388592 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" Nov 01 22:26:10.388611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.388604 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cztszk" event={"ID":"ccbe653a-5cd2-4a58-a4fd-20a905296842","Type":"ContainerDied","Data":"b5a89b3fad31b114b3a8f14ae92d8c764d84ceeb33302bed0345cc1546e4b0e2"} Nov 01 22:26:10.388756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.388636 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5a89b3fad31b114b3a8f14ae92d8c764d84ceeb33302bed0345cc1546e4b0e2" Nov 01 22:26:10.390774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.390752 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" Nov 01 22:26:10.390890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.390753 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q9686" event={"ID":"5f1260f7-c4b3-415a-ac3a-eb348619ed77","Type":"ContainerDied","Data":"2b746fdf9832475cbbb6472498b3de9de612ba83abdf124a0a9b0b7fb9cce08e"} Nov 01 22:26:10.390959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.390887 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b746fdf9832475cbbb6472498b3de9de612ba83abdf124a0a9b0b7fb9cce08e" Nov 01 22:26:10.393069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.393050 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" Nov 01 22:26:10.393228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.393201 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644hwf5b" event={"ID":"56d72c33-c1de-496e-8707-b6c742c724bc","Type":"ContainerDied","Data":"879e3d65da28a5f601be13755d466c27f2d23339c86397ea8002aa49a2b69aad"} Nov 01 22:26:10.393354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.393245 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="879e3d65da28a5f601be13755d466c27f2d23339c86397ea8002aa49a2b69aad" Nov 01 22:26:10.395752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.395733 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" event={"ID":"19a5da99-824a-4f97-a26c-c6507e083cff","Type":"ContainerDied","Data":"534396c6046902423e40c6ecfe3124dc9178524e30af4bffbf22985e62ae8cb4"} Nov 01 22:26:10.395859 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.395767 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="534396c6046902423e40c6ecfe3124dc9178524e30af4bffbf22985e62ae8cb4" Nov 01 22:26:10.395859 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:10.395787 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715gmzw" Nov 01 22:26:15.585304 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:26:15.585269 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:26:16.291392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291359 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz"] Nov 01 22:26:16.291633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291622 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerName="pull" Nov 01 22:26:16.291677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291634 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerName="pull" Nov 01 22:26:16.291677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291648 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerName="extract" Nov 01 22:26:16.291677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291653 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerName="extract" Nov 01 22:26:16.291677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291661 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerName="extract" Nov 01 22:26:16.291677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291665 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerName="extract" Nov 01 22:26:16.291677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291673 6287 cpu_manager.go:414] "containerMap: removing container" podUID="56d72c33-c1de-496e-8707-b6c742c724bc" containerName="extract" Nov 01 22:26:16.291677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291678 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d72c33-c1de-496e-8707-b6c742c724bc" containerName="extract" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291686 6287 cpu_manager.go:414] "containerMap: removing container" podUID="17b72589-a7e2-45b7-a463-a51979d10858" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291694 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b72589-a7e2-45b7-a463-a51979d10858" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291700 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291704 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291711 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291715 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291721 6287 cpu_manager.go:414] "containerMap: removing container" podUID="19a5da99-824a-4f97-a26c-c6507e083cff" containerName="extract" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291725 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a5da99-824a-4f97-a26c-c6507e083cff" containerName="extract" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291732 6287 cpu_manager.go:414] "containerMap: removing container" podUID="56d72c33-c1de-496e-8707-b6c742c724bc" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291736 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d72c33-c1de-496e-8707-b6c742c724bc" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291742 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291746 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291752 6287 cpu_manager.go:414] "containerMap: removing container" podUID="19a5da99-824a-4f97-a26c-c6507e083cff" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291756 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a5da99-824a-4f97-a26c-c6507e083cff" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291764 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerName="extract" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291768 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerName="extract" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291774 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerName="extract" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291778 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerName="extract" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291784 6287 cpu_manager.go:414] "containerMap: removing container" podUID="19a5da99-824a-4f97-a26c-c6507e083cff" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291789 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a5da99-824a-4f97-a26c-c6507e083cff" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291797 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291801 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291808 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291812 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291818 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291822 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291828 6287 cpu_manager.go:414] "containerMap: removing container" podUID="17b72589-a7e2-45b7-a463-a51979d10858" containerName="extract" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291833 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b72589-a7e2-45b7-a463-a51979d10858" containerName="extract" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291839 6287 cpu_manager.go:414] "containerMap: removing container" podUID="17b72589-a7e2-45b7-a463-a51979d10858" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291844 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b72589-a7e2-45b7-a463-a51979d10858" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291853 6287 cpu_manager.go:414] "containerMap: removing container" podUID="56d72c33-c1de-496e-8707-b6c742c724bc" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291856 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d72c33-c1de-496e-8707-b6c742c724bc" containerName="pull" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291862 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerName="util" Nov 01 22:26:16.291866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291867 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerName="util" Nov 01 22:26:16.292777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291906 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d72c33-c1de-496e-8707-b6c742c724bc" containerName="extract" Nov 01 22:26:16.292777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291913 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a5da99-824a-4f97-a26c-c6507e083cff" containerName="extract" Nov 01 22:26:16.292777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291920 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f5a86b-b367-4d7e-986a-c27bd976ca18" containerName="extract" Nov 01 22:26:16.292777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291925 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74ac2cd-b340-4269-9285-133c9b5c22bd" containerName="extract" Nov 01 22:26:16.292777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291931 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b72589-a7e2-45b7-a463-a51979d10858" containerName="extract" Nov 01 22:26:16.292777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291938 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1260f7-c4b3-415a-ac3a-eb348619ed77" containerName="extract" Nov 01 22:26:16.292777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.291943 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbe653a-5cd2-4a58-a4fd-20a905296842" containerName="extract" Nov 01 22:26:16.294583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.294566 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" Nov 01 22:26:16.302505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.302413 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 22:26:16.302505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.302466 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 22:26:16.302505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.302417 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-56l7j\"" Nov 01 22:26:16.302505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.302481 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 22:26:16.303673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.303649 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz"] Nov 01 22:26:16.340990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.340962 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfzl\" (UniqueName: \"kubernetes.io/projected/c74aadae-e2f9-47f6-b533-88919cce7c90-kube-api-access-pkfzl\") pod \"dns-operator-controller-manager-8557cbdb84-pv9xz\" (UID: \"c74aadae-e2f9-47f6-b533-88919cce7c90\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" Nov 01 22:26:16.441558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.441530 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfzl\" (UniqueName: \"kubernetes.io/projected/c74aadae-e2f9-47f6-b533-88919cce7c90-kube-api-access-pkfzl\") pod \"dns-operator-controller-manager-8557cbdb84-pv9xz\" (UID: \"c74aadae-e2f9-47f6-b533-88919cce7c90\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" Nov 01 22:26:16.466050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.466027 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfzl\" (UniqueName: \"kubernetes.io/projected/c74aadae-e2f9-47f6-b533-88919cce7c90-kube-api-access-pkfzl\") pod \"dns-operator-controller-manager-8557cbdb84-pv9xz\" (UID: \"c74aadae-e2f9-47f6-b533-88919cce7c90\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" Nov 01 22:26:16.607498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.607432 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" Nov 01 22:26:16.876637 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:16.876609 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz"] Nov 01 22:26:16.879004 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:26:16.878975 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74aadae_e2f9_47f6_b533_88919cce7c90.slice/crio-08b65f99eff631f98c06a981590771001c0e7678a81fa3c7606ac38f02f3a1c9 WatchSource:0}: Error finding container 08b65f99eff631f98c06a981590771001c0e7678a81fa3c7606ac38f02f3a1c9: Status 404 returned error can't find the container with id 08b65f99eff631f98c06a981590771001c0e7678a81fa3c7606ac38f02f3a1c9 Nov 01 22:26:17.427424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:17.427387 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" event={"ID":"c74aadae-e2f9-47f6-b533-88919cce7c90","Type":"ContainerStarted","Data":"08b65f99eff631f98c06a981590771001c0e7678a81fa3c7606ac38f02f3a1c9"} Nov 01 22:26:20.314959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.314928 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx"] Nov 01 22:26:20.316942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.316925 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:26:20.328198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.328178 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lc64k\"" Nov 01 22:26:20.330048 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.330023 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx"] Nov 01 22:26:20.380216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.380192 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvmc\" (UniqueName: \"kubernetes.io/projected/bffc1a34-b42b-45e6-9350-31089c369391-kube-api-access-4mvmc\") pod \"limitador-operator-controller-manager-5778bb4449-z26xx\" (UID: \"bffc1a34-b42b-45e6-9350-31089c369391\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:26:20.442847 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.442817 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" event={"ID":"c74aadae-e2f9-47f6-b533-88919cce7c90","Type":"ContainerStarted","Data":"2d3c35e358180915eadca3d53556f31d601190e06962ef16cbd91106c2734ef2"} Nov 01 22:26:20.442965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.442887 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" Nov 01 22:26:20.472741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.472698 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" podStartSLOduration=1.420590963 podStartE2EDuration="4.472686337s" totalImagesPullingTime="3.052095374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:20.471509505 +0000 UTC m=+193.120008737" watchObservedRunningTime="2026-11-01 22:26:20.472686337 +0000 UTC m=+193.121185572" Nov 01 22:26:20.481391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.481358 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvmc\" (UniqueName: \"kubernetes.io/projected/bffc1a34-b42b-45e6-9350-31089c369391-kube-api-access-4mvmc\") pod \"limitador-operator-controller-manager-5778bb4449-z26xx\" (UID: \"bffc1a34-b42b-45e6-9350-31089c369391\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:26:20.493251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.493225 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvmc\" (UniqueName: \"kubernetes.io/projected/bffc1a34-b42b-45e6-9350-31089c369391-kube-api-access-4mvmc\") pod \"limitador-operator-controller-manager-5778bb4449-z26xx\" (UID: \"bffc1a34-b42b-45e6-9350-31089c369391\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:26:20.627387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.627324 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:26:20.900866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:20.900840 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx"] Nov 01 22:26:20.903076 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:26:20.903047 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffc1a34_b42b_45e6_9350_31089c369391.slice/crio-c86b4ab85c69db858767d879cdac96c417dd7dc2621c96762d373098c0ef28a7 WatchSource:0}: Error finding container c86b4ab85c69db858767d879cdac96c417dd7dc2621c96762d373098c0ef28a7: Status 404 returned error can't find the container with id c86b4ab85c69db858767d879cdac96c417dd7dc2621c96762d373098c0ef28a7 Nov 01 22:26:21.449308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:21.449274 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" event={"ID":"bffc1a34-b42b-45e6-9350-31089c369391","Type":"ContainerStarted","Data":"c86b4ab85c69db858767d879cdac96c417dd7dc2621c96762d373098c0ef28a7"} Nov 01 22:26:22.664185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.664149 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd"] Nov 01 22:26:22.666863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.666838 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" Nov 01 22:26:22.673947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.673919 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9cj9c\"" Nov 01 22:26:22.676939 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.676923 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:26:22.677034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.676947 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 22:26:22.686498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.686469 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd"] Nov 01 22:26:22.802999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.802959 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-994p9\" (UniqueName: \"kubernetes.io/projected/838c5b5f-08e0-46a8-a913-74fb85ac37c1-kube-api-access-994p9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n25hd\" (UID: \"838c5b5f-08e0-46a8-a913-74fb85ac37c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" Nov 01 22:26:22.803207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.803078 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/838c5b5f-08e0-46a8-a913-74fb85ac37c1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n25hd\" (UID: \"838c5b5f-08e0-46a8-a913-74fb85ac37c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" Nov 01 22:26:22.903823 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.903790 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/838c5b5f-08e0-46a8-a913-74fb85ac37c1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n25hd\" (UID: \"838c5b5f-08e0-46a8-a913-74fb85ac37c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" Nov 01 22:26:22.903972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.903854 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-994p9\" (UniqueName: \"kubernetes.io/projected/838c5b5f-08e0-46a8-a913-74fb85ac37c1-kube-api-access-994p9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n25hd\" (UID: \"838c5b5f-08e0-46a8-a913-74fb85ac37c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" Nov 01 22:26:22.904193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.904175 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/838c5b5f-08e0-46a8-a913-74fb85ac37c1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n25hd\" (UID: \"838c5b5f-08e0-46a8-a913-74fb85ac37c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" Nov 01 22:26:22.914911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.914852 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-994p9\" (UniqueName: \"kubernetes.io/projected/838c5b5f-08e0-46a8-a913-74fb85ac37c1-kube-api-access-994p9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n25hd\" (UID: \"838c5b5f-08e0-46a8-a913-74fb85ac37c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" Nov 01 22:26:22.979229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:22.979204 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" Nov 01 22:26:23.255829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:23.255654 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd"] Nov 01 22:26:23.258388 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:26:23.258355 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838c5b5f_08e0_46a8_a913_74fb85ac37c1.slice/crio-2067d4cb0df7b2449d160c63b9bec206e4a40c9c1b0bd5c26a6ff1e68a03ae77 WatchSource:0}: Error finding container 2067d4cb0df7b2449d160c63b9bec206e4a40c9c1b0bd5c26a6ff1e68a03ae77: Status 404 returned error can't find the container with id 2067d4cb0df7b2449d160c63b9bec206e4a40c9c1b0bd5c26a6ff1e68a03ae77 Nov 01 22:26:23.459575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:23.459540 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" event={"ID":"838c5b5f-08e0-46a8-a913-74fb85ac37c1","Type":"ContainerStarted","Data":"2067d4cb0df7b2449d160c63b9bec206e4a40c9c1b0bd5c26a6ff1e68a03ae77"} Nov 01 22:26:24.466145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:24.466093 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" event={"ID":"bffc1a34-b42b-45e6-9350-31089c369391","Type":"ContainerStarted","Data":"e00dca99dfb032a88390430dfd62c2779ffa261d7fa9679687fa2c2780b37b5b"} Nov 01 22:26:24.466637 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:24.466231 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:26:24.490146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:24.490084 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" podStartSLOduration=1.9130011279999999 podStartE2EDuration="4.49007203s" totalImagesPullingTime="2.577070902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:24.489703998 +0000 UTC m=+197.138203234" watchObservedRunningTime="2026-11-01 22:26:24.49007203 +0000 UTC m=+197.138571261" Nov 01 22:26:25.471454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:25.471366 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" event={"ID":"838c5b5f-08e0-46a8-a913-74fb85ac37c1","Type":"ContainerStarted","Data":"ba675b8b4293603b7c305c9957b520759966b678f9f9dd5940ee6734426b5426"} Nov 01 22:26:25.493424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:25.493378 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n25hd" podStartSLOduration=1.542663034 podStartE2EDuration="3.493366368s" totalImagesPullingTime="1.950703334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:25.491024001 +0000 UTC m=+198.139523237" watchObservedRunningTime="2026-11-01 22:26:25.493366368 +0000 UTC m=+198.141865596" Nov 01 22:26:26.606934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.606897 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq"] Nov 01 22:26:26.610529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.610510 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.643904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.643875 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq"] Nov 01 22:26:26.697366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.697339 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 22:26:26.697506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.697488 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-fc42r\"" Nov 01 22:26:26.697558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.697491 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 22:26:26.697558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.697511 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 22:26:26.697631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.697550 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 22:26:26.697631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.697593 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 22:26:26.736642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.736610 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50a9903-80f6-457c-bcf9-9838a3cbdd64-tmp\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.736782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.736658 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c50a9903-80f6-457c-bcf9-9838a3cbdd64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.736782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.736690 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c50a9903-80f6-457c-bcf9-9838a3cbdd64-manifests\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.736873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.736859 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p974\" (UniqueName: \"kubernetes.io/projected/c50a9903-80f6-457c-bcf9-9838a3cbdd64-kube-api-access-2p974\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.736916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.736880 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c50a9903-80f6-457c-bcf9-9838a3cbdd64-metrics-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.736916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.736902 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c50a9903-80f6-457c-bcf9-9838a3cbdd64-webhook-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.837932 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.837898 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c50a9903-80f6-457c-bcf9-9838a3cbdd64-manifests\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.838076 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.837943 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p974\" (UniqueName: \"kubernetes.io/projected/c50a9903-80f6-457c-bcf9-9838a3cbdd64-kube-api-access-2p974\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.838076 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.837964 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c50a9903-80f6-457c-bcf9-9838a3cbdd64-metrics-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.838194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.838087 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c50a9903-80f6-457c-bcf9-9838a3cbdd64-webhook-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.838233 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.838193 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50a9903-80f6-457c-bcf9-9838a3cbdd64-tmp\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.838283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.838245 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c50a9903-80f6-457c-bcf9-9838a3cbdd64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.838375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.838356 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c50a9903-80f6-457c-bcf9-9838a3cbdd64-manifests\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.840443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.840412 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50a9903-80f6-457c-bcf9-9838a3cbdd64-tmp\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.840551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.840462 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c50a9903-80f6-457c-bcf9-9838a3cbdd64-metrics-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.840551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.840542 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c50a9903-80f6-457c-bcf9-9838a3cbdd64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.840632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.840580 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c50a9903-80f6-457c-bcf9-9838a3cbdd64-webhook-cert\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.852274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.852254 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p974\" (UniqueName: \"kubernetes.io/projected/c50a9903-80f6-457c-bcf9-9838a3cbdd64-kube-api-access-2p974\") pod \"opendatahub-operator-controller-manager-8487c6b6d6-r86kq\" (UID: \"c50a9903-80f6-457c-bcf9-9838a3cbdd64\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:26.925379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:26.925327 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:27.148716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.148679 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq"] Nov 01 22:26:27.152923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.152905 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:27.162131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.162096 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 22:26:27.162249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.162147 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 22:26:27.162249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.162190 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mxbxs\"" Nov 01 22:26:27.169372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.169348 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq"] Nov 01 22:26:27.225273 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.225223 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq"] Nov 01 22:26:27.228962 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:26:27.228932 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50a9903_80f6_457c_bcf9_9838a3cbdd64.slice/crio-c707e1384d1dc0599becc708b335f570e576558ba72b0f589407c1b290bbd8e3 WatchSource:0}: Error finding container c707e1384d1dc0599becc708b335f570e576558ba72b0f589407c1b290bbd8e3: Status 404 returned error can't find the container with id c707e1384d1dc0599becc708b335f570e576558ba72b0f589407c1b290bbd8e3 Nov 01 22:26:27.241341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.241320 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f95020b-ad58-422a-8908-3d46472f9d35-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-h4rbq\" (UID: \"1f95020b-ad58-422a-8908-3d46472f9d35\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:27.241434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.241420 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gwl\" (UniqueName: \"kubernetes.io/projected/1f95020b-ad58-422a-8908-3d46472f9d35-kube-api-access-q2gwl\") pod \"cert-manager-webhook-5dbbbf75cc-h4rbq\" (UID: \"1f95020b-ad58-422a-8908-3d46472f9d35\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:27.342650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.342624 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gwl\" (UniqueName: \"kubernetes.io/projected/1f95020b-ad58-422a-8908-3d46472f9d35-kube-api-access-q2gwl\") pod \"cert-manager-webhook-5dbbbf75cc-h4rbq\" (UID: \"1f95020b-ad58-422a-8908-3d46472f9d35\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:27.342791 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.342671 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f95020b-ad58-422a-8908-3d46472f9d35-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-h4rbq\" (UID: \"1f95020b-ad58-422a-8908-3d46472f9d35\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:27.350958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.350934 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f95020b-ad58-422a-8908-3d46472f9d35-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-h4rbq\" (UID: \"1f95020b-ad58-422a-8908-3d46472f9d35\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:27.351068 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.350986 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gwl\" (UniqueName: \"kubernetes.io/projected/1f95020b-ad58-422a-8908-3d46472f9d35-kube-api-access-q2gwl\") pod \"cert-manager-webhook-5dbbbf75cc-h4rbq\" (UID: \"1f95020b-ad58-422a-8908-3d46472f9d35\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:27.480403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.480332 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:27.481154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.481131 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" event={"ID":"c50a9903-80f6-457c-bcf9-9838a3cbdd64","Type":"ContainerStarted","Data":"c707e1384d1dc0599becc708b335f570e576558ba72b0f589407c1b290bbd8e3"} Nov 01 22:26:27.811504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:27.811479 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq"] Nov 01 22:26:27.813608 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:26:27.813582 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f95020b_ad58_422a_8908_3d46472f9d35.slice/crio-34cef1f4298193d4fa301ee1b823834a1528675151500d1c52fc11c8196391b8 WatchSource:0}: Error finding container 34cef1f4298193d4fa301ee1b823834a1528675151500d1c52fc11c8196391b8: Status 404 returned error can't find the container with id 34cef1f4298193d4fa301ee1b823834a1528675151500d1c52fc11c8196391b8 Nov 01 22:26:28.492167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:28.492105 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" event={"ID":"1f95020b-ad58-422a-8908-3d46472f9d35","Type":"ContainerStarted","Data":"34cef1f4298193d4fa301ee1b823834a1528675151500d1c52fc11c8196391b8"} Nov 01 22:26:29.443763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.442106 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7"] Nov 01 22:26:29.451366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.451341 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" Nov 01 22:26:29.460365 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.459976 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7"] Nov 01 22:26:29.514250 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.514218 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-z7clj\"" Nov 01 22:26:29.562859 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.562827 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6503191-bfb0-4745-bf24-3fb040974bba-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qscq7\" (UID: \"e6503191-bfb0-4745-bf24-3fb040974bba\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" Nov 01 22:26:29.563024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.562933 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbn2\" (UniqueName: \"kubernetes.io/projected/e6503191-bfb0-4745-bf24-3fb040974bba-kube-api-access-rxbn2\") pod \"cert-manager-cainjector-5c57bbd4fc-qscq7\" (UID: \"e6503191-bfb0-4745-bf24-3fb040974bba\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" Nov 01 22:26:29.664254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.664216 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbn2\" (UniqueName: \"kubernetes.io/projected/e6503191-bfb0-4745-bf24-3fb040974bba-kube-api-access-rxbn2\") pod \"cert-manager-cainjector-5c57bbd4fc-qscq7\" (UID: \"e6503191-bfb0-4745-bf24-3fb040974bba\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" Nov 01 22:26:29.664439 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.664337 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6503191-bfb0-4745-bf24-3fb040974bba-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qscq7\" (UID: \"e6503191-bfb0-4745-bf24-3fb040974bba\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" Nov 01 22:26:29.675953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.675919 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbn2\" (UniqueName: \"kubernetes.io/projected/e6503191-bfb0-4745-bf24-3fb040974bba-kube-api-access-rxbn2\") pod \"cert-manager-cainjector-5c57bbd4fc-qscq7\" (UID: \"e6503191-bfb0-4745-bf24-3fb040974bba\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" Nov 01 22:26:29.685902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.685801 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6503191-bfb0-4745-bf24-3fb040974bba-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qscq7\" (UID: \"e6503191-bfb0-4745-bf24-3fb040974bba\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" Nov 01 22:26:29.767943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:29.767912 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" Nov 01 22:26:30.059000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:30.058972 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7"] Nov 01 22:26:30.506072 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:30.506041 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" event={"ID":"e6503191-bfb0-4745-bf24-3fb040974bba","Type":"ContainerStarted","Data":"0bda4a7dc40c321876f8c7f710cf94e3a739074367eae2aa23fba7f03e2ea642"} Nov 01 22:26:31.451773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:31.451736 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" Nov 01 22:26:31.513287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:31.513197 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" event={"ID":"1f95020b-ad58-422a-8908-3d46472f9d35","Type":"ContainerStarted","Data":"b9065e2b5543291bd26612815668efb35957c83ac78e549fbc45c51ceb708248"} Nov 01 22:26:31.516781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:31.516750 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" event={"ID":"c50a9903-80f6-457c-bcf9-9838a3cbdd64","Type":"ContainerStarted","Data":"d782ba4966622b14bd2309037dafe70bb069555f4f8ca49325fe46d3ec64a8f3"} Nov 01 22:26:32.522870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:32.522840 6287 generic.go:356] "Generic (PLEG): container finished" podID="c50a9903-80f6-457c-bcf9-9838a3cbdd64" containerID="d782ba4966622b14bd2309037dafe70bb069555f4f8ca49325fe46d3ec64a8f3" exitCode=0 Nov 01 22:26:32.523274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:32.522930 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" event={"ID":"c50a9903-80f6-457c-bcf9-9838a3cbdd64","Type":"ContainerDied","Data":"d782ba4966622b14bd2309037dafe70bb069555f4f8ca49325fe46d3ec64a8f3"} Nov 01 22:26:32.523274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:32.523086 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:32.617797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:32.617736 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podStartSLOduration=2.008579674 podStartE2EDuration="5.617717895s" totalImagesPullingTime="3.609138221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:32.61747752 +0000 UTC m=+205.265976756" watchObservedRunningTime="2026-11-01 22:26:32.617717895 +0000 UTC m=+205.266217131" Nov 01 22:26:33.529681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:33.529585 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" event={"ID":"e6503191-bfb0-4745-bf24-3fb040974bba","Type":"ContainerStarted","Data":"30ff61f8208a7e940bfe26a498c60eba3296709d7c00c3c2b25a807dc6242b28"} Nov 01 22:26:33.531811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:33.531779 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" event={"ID":"c50a9903-80f6-457c-bcf9-9838a3cbdd64","Type":"ContainerStarted","Data":"8ba30928825707a504b3e73c515a802dad8af175f3c1af38773132ec85e2dc7f"} Nov 01 22:26:33.531952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:33.531858 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:33.546436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:33.546380 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qscq7" podStartSLOduration=1.345384928 podStartE2EDuration="4.54636272s" totalImagesPullingTime="3.200977792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:33.545349442 +0000 UTC m=+206.193848678" watchObservedRunningTime="2026-11-01 22:26:33.54636272 +0000 UTC m=+206.194861978" Nov 01 22:26:33.585781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:33.585730 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" podStartSLOduration=2.605822672 podStartE2EDuration="7.58571507s" totalImagesPullingTime="4.414347799s" totalInitContainerRuntime="565.544599ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:33.582072064 +0000 UTC m=+206.230571299" watchObservedRunningTime="2026-11-01 22:26:33.58571507 +0000 UTC m=+206.234214298" Nov 01 22:26:35.473932 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:35.473903 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:26:38.534623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:38.534594 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:26:44.538149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:44.538096 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" Nov 01 22:26:45.586524 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:26:45.586490 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:26:47.354137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.354093 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-vp7k9"] Nov 01 22:26:47.357286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.357268 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vp7k9" Nov 01 22:26:47.365903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.365879 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2sdwq\"" Nov 01 22:26:47.368336 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.368309 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vp7k9"] Nov 01 22:26:47.514953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.514919 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd-bound-sa-token\") pod \"cert-manager-597b95cb84-vp7k9\" (UID: \"2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd\") " pod="cert-manager/cert-manager-597b95cb84-vp7k9" Nov 01 22:26:47.515177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.514961 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpq27\" (UniqueName: \"kubernetes.io/projected/2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd-kube-api-access-xpq27\") pod \"cert-manager-597b95cb84-vp7k9\" (UID: \"2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd\") " pod="cert-manager/cert-manager-597b95cb84-vp7k9" Nov 01 22:26:47.615908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.615818 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd-bound-sa-token\") pod \"cert-manager-597b95cb84-vp7k9\" (UID: \"2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd\") " pod="cert-manager/cert-manager-597b95cb84-vp7k9" Nov 01 22:26:47.615908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.615865 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpq27\" (UniqueName: \"kubernetes.io/projected/2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd-kube-api-access-xpq27\") pod \"cert-manager-597b95cb84-vp7k9\" (UID: \"2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd\") " pod="cert-manager/cert-manager-597b95cb84-vp7k9" Nov 01 22:26:47.623776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.623747 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd-bound-sa-token\") pod \"cert-manager-597b95cb84-vp7k9\" (UID: \"2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd\") " pod="cert-manager/cert-manager-597b95cb84-vp7k9" Nov 01 22:26:47.624155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.624137 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpq27\" (UniqueName: \"kubernetes.io/projected/2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd-kube-api-access-xpq27\") pod \"cert-manager-597b95cb84-vp7k9\" (UID: \"2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd\") " pod="cert-manager/cert-manager-597b95cb84-vp7k9" Nov 01 22:26:47.669624 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.669594 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vp7k9" Nov 01 22:26:47.949657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:47.949629 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vp7k9"] Nov 01 22:26:47.951969 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:26:47.951938 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a4d1303_cdef_4dc9_afc1_12c1af9e4dcd.slice/crio-419bcfd78fdf9964bf5346bbf674dfdf23640dfe0edbf5f58db6449d89d82648 WatchSource:0}: Error finding container 419bcfd78fdf9964bf5346bbf674dfdf23640dfe0edbf5f58db6449d89d82648: Status 404 returned error can't find the container with id 419bcfd78fdf9964bf5346bbf674dfdf23640dfe0edbf5f58db6449d89d82648 Nov 01 22:26:48.596715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:48.596678 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vp7k9" event={"ID":"2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd","Type":"ContainerStarted","Data":"cd33d3624be7c876561df368a208067c3164241c4a9d28b6a1572d452606ece1"} Nov 01 22:26:48.596715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:48.596726 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vp7k9" event={"ID":"2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd","Type":"ContainerStarted","Data":"419bcfd78fdf9964bf5346bbf674dfdf23640dfe0edbf5f58db6449d89d82648"} Nov 01 22:26:48.617443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:48.617399 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-vp7k9" podStartSLOduration=1.6173882960000001 podStartE2EDuration="1.617388296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:48.616233805 +0000 UTC m=+221.264733040" watchObservedRunningTime="2026-11-01 22:26:48.617388296 +0000 UTC m=+221.265887531" Nov 01 22:26:52.232575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.232541 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79d9489c85-swvcv"] Nov 01 22:26:52.240578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.240552 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.246008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.245562 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79d9489c85-swvcv"] Nov 01 22:26:52.249484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.249460 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6bl\" (UniqueName: \"kubernetes.io/projected/50a0b3cf-a357-4b31-8d99-65952baf4230-kube-api-access-xw6bl\") pod \"kube-auth-proxy-79d9489c85-swvcv\" (UID: \"50a0b3cf-a357-4b31-8d99-65952baf4230\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.249664 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.249539 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50a0b3cf-a357-4b31-8d99-65952baf4230-tmp\") pod \"kube-auth-proxy-79d9489c85-swvcv\" (UID: \"50a0b3cf-a357-4b31-8d99-65952baf4230\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.249760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.249677 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a0b3cf-a357-4b31-8d99-65952baf4230-tls-certs\") pod \"kube-auth-proxy-79d9489c85-swvcv\" (UID: \"50a0b3cf-a357-4b31-8d99-65952baf4230\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.256006 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.255976 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 22:26:52.256006 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.256002 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 22:26:52.256712 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.256693 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gj492\"" Nov 01 22:26:52.350439 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.350398 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a0b3cf-a357-4b31-8d99-65952baf4230-tls-certs\") pod \"kube-auth-proxy-79d9489c85-swvcv\" (UID: \"50a0b3cf-a357-4b31-8d99-65952baf4230\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.350632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.350490 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6bl\" (UniqueName: \"kubernetes.io/projected/50a0b3cf-a357-4b31-8d99-65952baf4230-kube-api-access-xw6bl\") pod \"kube-auth-proxy-79d9489c85-swvcv\" (UID: \"50a0b3cf-a357-4b31-8d99-65952baf4230\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.350632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.350559 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50a0b3cf-a357-4b31-8d99-65952baf4230-tmp\") pod \"kube-auth-proxy-79d9489c85-swvcv\" (UID: \"50a0b3cf-a357-4b31-8d99-65952baf4230\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.352989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.352957 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50a0b3cf-a357-4b31-8d99-65952baf4230-tmp\") pod \"kube-auth-proxy-79d9489c85-swvcv\" (UID: \"50a0b3cf-a357-4b31-8d99-65952baf4230\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.353249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.353229 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50a0b3cf-a357-4b31-8d99-65952baf4230-tls-certs\") pod \"kube-auth-proxy-79d9489c85-swvcv\" (UID: \"50a0b3cf-a357-4b31-8d99-65952baf4230\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.360009 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.359976 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6bl\" (UniqueName: \"kubernetes.io/projected/50a0b3cf-a357-4b31-8d99-65952baf4230-kube-api-access-xw6bl\") pod \"kube-auth-proxy-79d9489c85-swvcv\" (UID: \"50a0b3cf-a357-4b31-8d99-65952baf4230\") " pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.556494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.556457 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" Nov 01 22:26:52.843931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:52.843899 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79d9489c85-swvcv"] Nov 01 22:26:52.846289 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:26:52.846254 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a0b3cf_a357_4b31_8d99_65952baf4230.slice/crio-2428f8432c36eec65b334505b3c327ae8d150f5d30cd52f165c06f8ebc2d4a12 WatchSource:0}: Error finding container 2428f8432c36eec65b334505b3c327ae8d150f5d30cd52f165c06f8ebc2d4a12: Status 404 returned error can't find the container with id 2428f8432c36eec65b334505b3c327ae8d150f5d30cd52f165c06f8ebc2d4a12 Nov 01 22:26:53.624530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:53.624490 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" event={"ID":"50a0b3cf-a357-4b31-8d99-65952baf4230","Type":"ContainerStarted","Data":"2428f8432c36eec65b334505b3c327ae8d150f5d30cd52f165c06f8ebc2d4a12"} Nov 01 22:26:53.911294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:53.911212 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ctnfv"] Nov 01 22:26:53.915149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:53.915101 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" Nov 01 22:26:53.924596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:53.924538 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ctnfv"] Nov 01 22:26:53.956283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:53.956256 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-brn77\"" Nov 01 22:26:53.965836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:53.965800 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzmw\" (UniqueName: \"kubernetes.io/projected/dd6ba577-201a-4fca-b022-5306fe649a46-kube-api-access-slzmw\") pod \"authorino-57f7f8659f-ctnfv\" (UID: \"dd6ba577-201a-4fca-b022-5306fe649a46\") " pod="kuadrant-system/authorino-57f7f8659f-ctnfv" Nov 01 22:26:54.066734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.066694 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slzmw\" (UniqueName: \"kubernetes.io/projected/dd6ba577-201a-4fca-b022-5306fe649a46-kube-api-access-slzmw\") pod \"authorino-57f7f8659f-ctnfv\" (UID: \"dd6ba577-201a-4fca-b022-5306fe649a46\") " pod="kuadrant-system/authorino-57f7f8659f-ctnfv" Nov 01 22:26:54.079224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.079188 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzmw\" (UniqueName: \"kubernetes.io/projected/dd6ba577-201a-4fca-b022-5306fe649a46-kube-api-access-slzmw\") pod \"authorino-57f7f8659f-ctnfv\" (UID: \"dd6ba577-201a-4fca-b022-5306fe649a46\") " pod="kuadrant-system/authorino-57f7f8659f-ctnfv" Nov 01 22:26:54.233809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.233718 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" Nov 01 22:26:54.516207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.516180 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ctnfv"] Nov 01 22:26:54.524225 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:26:54.519182 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd6ba577_201a_4fca_b022_5306fe649a46.slice/crio-219df09f3cb02761b757afb6883c0bab43a1751f0a4374d15fccbbbb26fbe493 WatchSource:0}: Error finding container 219df09f3cb02761b757afb6883c0bab43a1751f0a4374d15fccbbbb26fbe493: Status 404 returned error can't find the container with id 219df09f3cb02761b757afb6883c0bab43a1751f0a4374d15fccbbbb26fbe493 Nov 01 22:26:54.633890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.633847 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" event={"ID":"dd6ba577-201a-4fca-b022-5306fe649a46","Type":"ContainerStarted","Data":"219df09f3cb02761b757afb6883c0bab43a1751f0a4374d15fccbbbb26fbe493"} Nov 01 22:26:54.936646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.936563 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx"] Nov 01 22:26:54.940442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.940418 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:54.949539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.949515 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 01 22:26:54.949797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.949772 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-x7l76\"" Nov 01 22:26:54.949904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.949876 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 01 22:26:54.949960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.949935 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx"] Nov 01 22:26:54.950002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.949958 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:26:54.950002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.949989 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 01 22:26:54.950085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.950015 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 01 22:26:54.976120 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.976086 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4fxd\" (UniqueName: \"kubernetes.io/projected/3633d07f-afe4-4477-bc86-d1881c6db757-kube-api-access-c4fxd\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:54.976310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.976180 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3633d07f-afe4-4477-bc86-d1881c6db757-cert\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:54.976310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.976263 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3633d07f-afe4-4477-bc86-d1881c6db757-metrics-certs\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:54.976310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:54.976292 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3633d07f-afe4-4477-bc86-d1881c6db757-manager-config\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.077557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.077515 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3633d07f-afe4-4477-bc86-d1881c6db757-metrics-certs\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.077753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.077573 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3633d07f-afe4-4477-bc86-d1881c6db757-manager-config\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.077753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.077649 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4fxd\" (UniqueName: \"kubernetes.io/projected/3633d07f-afe4-4477-bc86-d1881c6db757-kube-api-access-c4fxd\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.077753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.077699 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3633d07f-afe4-4477-bc86-d1881c6db757-cert\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.078515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.078420 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3633d07f-afe4-4477-bc86-d1881c6db757-manager-config\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.080572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.080545 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3633d07f-afe4-4477-bc86-d1881c6db757-metrics-certs\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.085608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.085586 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4fxd\" (UniqueName: \"kubernetes.io/projected/3633d07f-afe4-4477-bc86-d1881c6db757-kube-api-access-c4fxd\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.094537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.094517 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3633d07f-afe4-4477-bc86-d1881c6db757-cert\") pod \"jobset-controller-manager-76ffc984cf-cpvzx\" (UID: \"3633d07f-afe4-4477-bc86-d1881c6db757\") " pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.256729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.256699 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:26:55.550332 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.550309 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx"] Nov 01 22:26:55.552247 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:26:55.552219 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3633d07f_afe4_4477_bc86_d1881c6db757.slice/crio-60b05f04a4041928373c98670b3f95e4fb4da8b8ed1c1506df7edba4f28df164 WatchSource:0}: Error finding container 60b05f04a4041928373c98670b3f95e4fb4da8b8ed1c1506df7edba4f28df164: Status 404 returned error can't find the container with id 60b05f04a4041928373c98670b3f95e4fb4da8b8ed1c1506df7edba4f28df164 Nov 01 22:26:55.640470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.640430 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" event={"ID":"50a0b3cf-a357-4b31-8d99-65952baf4230","Type":"ContainerStarted","Data":"6d1186657ae83174ad68584e340317dd9ccda060dfec842f127b4322b0c8f9a2"} Nov 01 22:26:55.642177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.642052 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" event={"ID":"3633d07f-afe4-4477-bc86-d1881c6db757","Type":"ContainerStarted","Data":"60b05f04a4041928373c98670b3f95e4fb4da8b8ed1c1506df7edba4f28df164"} Nov 01 22:26:55.658952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:55.658903 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79d9489c85-swvcv" podStartSLOduration=1.229290056 podStartE2EDuration="3.658890135s" totalImagesPullingTime="2.429600079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:55.657267085 +0000 UTC m=+228.305766321" watchObservedRunningTime="2026-11-01 22:26:55.658890135 +0000 UTC m=+228.307389370" Nov 01 22:26:58.352351 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:58.352270 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ctnfv"] Nov 01 22:26:58.657150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:58.657053 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" event={"ID":"dd6ba577-201a-4fca-b022-5306fe649a46","Type":"ContainerStarted","Data":"7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046"} Nov 01 22:26:58.675799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:58.675746 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" podStartSLOduration=2.161648398 podStartE2EDuration="5.675733147s" totalImagesPullingTime="3.514084749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:58.674854518 +0000 UTC m=+231.323353753" watchObservedRunningTime="2026-11-01 22:26:58.675733147 +0000 UTC m=+231.324232381" Nov 01 22:26:59.661338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:59.661298 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" podUID="dd6ba577-201a-4fca-b022-5306fe649a46" containerName="authorino" containerID="cri-o://7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046" gracePeriod=30 Nov 01 22:26:59.918558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:26:59.918497 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" Nov 01 22:27:00.022283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.022256 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6ba577-201a-4fca-b022-5306fe649a46-kube-api-access-slzmw\" (UniqueName: \"kubernetes.io/projected/dd6ba577-201a-4fca-b022-5306fe649a46-kube-api-access-slzmw\") pod \"dd6ba577-201a-4fca-b022-5306fe649a46\" (UID: \"dd6ba577-201a-4fca-b022-5306fe649a46\") " Nov 01 22:27:00.027180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.027153 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6ba577-201a-4fca-b022-5306fe649a46-kube-api-access-slzmw" pod "dd6ba577-201a-4fca-b022-5306fe649a46" (UID: "dd6ba577-201a-4fca-b022-5306fe649a46"). InnerVolumeSpecName "kube-api-access-slzmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:27:00.123813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.123781 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slzmw\" (UniqueName: \"kubernetes.io/projected/dd6ba577-201a-4fca-b022-5306fe649a46-kube-api-access-slzmw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:27:00.666618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.666580 6287 generic.go:356] "Generic (PLEG): container finished" podID="dd6ba577-201a-4fca-b022-5306fe649a46" containerID="7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046" exitCode=0 Nov 01 22:27:00.667207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.666625 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" Nov 01 22:27:00.667207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.666654 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" event={"ID":"dd6ba577-201a-4fca-b022-5306fe649a46","Type":"ContainerDied","Data":"7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046"} Nov 01 22:27:00.667207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.666688 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ctnfv" event={"ID":"dd6ba577-201a-4fca-b022-5306fe649a46","Type":"ContainerDied","Data":"219df09f3cb02761b757afb6883c0bab43a1751f0a4374d15fccbbbb26fbe493"} Nov 01 22:27:00.667207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.666708 6287 scope.go:122] "RemoveContainer" containerID="7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046" Nov 01 22:27:00.668376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.668346 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" event={"ID":"3633d07f-afe4-4477-bc86-d1881c6db757","Type":"ContainerStarted","Data":"8cf633fd4f8871812a1d0ccb3e8615c1718422aa333a19aa882df9e98eab411c"} Nov 01 22:27:00.668868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.668848 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:27:00.682057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.682037 6287 scope.go:122] "RemoveContainer" containerID="7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046" Nov 01 22:27:00.682342 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:27:00.682324 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046\": container with ID starting with 7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046 not found: ID does not exist" containerID="7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046" Nov 01 22:27:00.682435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.682349 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046"} err="failed to get container status \"7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046\": rpc error: code = NotFound desc = could not find container \"7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046\": container with ID starting with 7ba0c9a72906adb1fc0e2537caed45ec06ce0e9a604fd06ae87678a87d5f6046 not found: ID does not exist" Nov 01 22:27:00.686729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.686692 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" podStartSLOduration=2.219091168 podStartE2EDuration="6.686679108s" totalImagesPullingTime="4.46758794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:00.685476799 +0000 UTC m=+233.333976036" watchObservedRunningTime="2026-11-01 22:27:00.686679108 +0000 UTC m=+233.335178342" Nov 01 22:27:00.699710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.699688 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ctnfv"] Nov 01 22:27:00.703305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:00.703287 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ctnfv"] Nov 01 22:27:01.468747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:01.468704 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6ba577-201a-4fca-b022-5306fe649a46" path="/var/lib/kubelet/pods/dd6ba577-201a-4fca-b022-5306fe649a46/volumes" Nov 01 22:27:12.683981 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:27:12.683950 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:27:15.586305 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:27:15.586270 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:27:45.585981 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:27:45.585937 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:28:15.585676 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:28:15.585635 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:28:32.113141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:32.113090 6287 generic.go:356] "Generic (PLEG): container finished" podID="71935876-95e7-4691-bedf-e040f4cfec12" containerID="2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c" exitCode=0 Nov 01 22:28:32.113605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:32.113145 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" event={"ID":"71935876-95e7-4691-bedf-e040f4cfec12","Type":"ContainerDied","Data":"2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c"} Nov 01 22:28:32.113605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:32.113519 6287 scope.go:122] "RemoveContainer" containerID="2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c" Nov 01 22:28:32.114678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:32.114648 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:28:32.114754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:32.114740 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:28:32.114797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:32.114787 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:28:32.146312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:32.146272 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:28:33.119199 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.119157 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" event={"ID":"71935876-95e7-4691-bedf-e040f4cfec12","Type":"ContainerStarted","Data":"bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5"} Nov 01 22:28:33.450962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.450880 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvfz7"] Nov 01 22:28:33.451272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.451257 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dd6ba577-201a-4fca-b022-5306fe649a46" containerName="authorino" Nov 01 22:28:33.451326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.451273 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6ba577-201a-4fca-b022-5306fe649a46" containerName="authorino" Nov 01 22:28:33.451371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.451345 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6ba577-201a-4fca-b022-5306fe649a46" containerName="authorino" Nov 01 22:28:33.454306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.454291 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wvfz7" Nov 01 22:28:33.462268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.462235 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-brn77\"" Nov 01 22:28:33.468576 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.468552 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvfz7"] Nov 01 22:28:33.495239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.495194 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:28:33.496267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.496226 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-87hds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 22:28:33.496371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.496311 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 22:28:33.499429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.499406 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvfz7"] Nov 01 22:28:33.499609 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:28:33.499591 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2tt72], unattached volumes=[], failed to process volumes=[kube-api-access-2tt72]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-wvfz7" podUID="1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3" Nov 01 22:28:33.613322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.613278 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tt72\" (UniqueName: \"kubernetes.io/projected/1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3-kube-api-access-2tt72\") pod \"authorino-6f6944bcdd-wvfz7\" (UID: \"1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3\") " pod="kuadrant-system/authorino-6f6944bcdd-wvfz7" Nov 01 22:28:33.714515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.714424 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tt72\" (UniqueName: \"kubernetes.io/projected/1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3-kube-api-access-2tt72\") pod \"authorino-6f6944bcdd-wvfz7\" (UID: \"1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3\") " pod="kuadrant-system/authorino-6f6944bcdd-wvfz7" Nov 01 22:28:33.722084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.722062 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tt72\" (UniqueName: \"kubernetes.io/projected/1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3-kube-api-access-2tt72\") pod \"authorino-6f6944bcdd-wvfz7\" (UID: \"1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3\") " pod="kuadrant-system/authorino-6f6944bcdd-wvfz7" Nov 01 22:28:33.824950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.824916 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-875bcff54-bhskn"] Nov 01 22:28:33.828283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.828260 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:28:33.835307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.835216 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 22:28:33.838319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.838295 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-875bcff54-bhskn"] Nov 01 22:28:33.916674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.916622 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvm65\" (UniqueName: \"kubernetes.io/projected/02318c1e-d93c-451c-88e4-5a09a1502ced-kube-api-access-kvm65\") pod \"authorino-875bcff54-bhskn\" (UID: \"02318c1e-d93c-451c-88e4-5a09a1502ced\") " pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:28:33.916847 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:33.916741 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02318c1e-d93c-451c-88e4-5a09a1502ced-tls-cert\") pod \"authorino-875bcff54-bhskn\" (UID: \"02318c1e-d93c-451c-88e4-5a09a1502ced\") " pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:28:34.017945 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.017905 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvm65\" (UniqueName: \"kubernetes.io/projected/02318c1e-d93c-451c-88e4-5a09a1502ced-kube-api-access-kvm65\") pod \"authorino-875bcff54-bhskn\" (UID: \"02318c1e-d93c-451c-88e4-5a09a1502ced\") " pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:28:34.018156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.017973 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02318c1e-d93c-451c-88e4-5a09a1502ced-tls-cert\") pod \"authorino-875bcff54-bhskn\" (UID: \"02318c1e-d93c-451c-88e4-5a09a1502ced\") " pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:28:34.020607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.020581 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02318c1e-d93c-451c-88e4-5a09a1502ced-tls-cert\") pod \"authorino-875bcff54-bhskn\" (UID: \"02318c1e-d93c-451c-88e4-5a09a1502ced\") " pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:28:34.026308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.026281 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvm65\" (UniqueName: \"kubernetes.io/projected/02318c1e-d93c-451c-88e4-5a09a1502ced-kube-api-access-kvm65\") pod \"authorino-875bcff54-bhskn\" (UID: \"02318c1e-d93c-451c-88e4-5a09a1502ced\") " pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:28:34.123398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.123368 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wvfz7" Nov 01 22:28:34.130423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.130399 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wvfz7" Nov 01 22:28:34.141155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.141135 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:28:34.219703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.219668 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3-kube-api-access-2tt72\" (UniqueName: \"kubernetes.io/projected/1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3-kube-api-access-2tt72\") pod \"1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3\" (UID: \"1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3\") " Nov 01 22:28:34.223560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.223529 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3-kube-api-access-2tt72" pod "1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3" (UID: "1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3"). InnerVolumeSpecName "kube-api-access-2tt72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:28:34.320955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.320876 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tt72\" (UniqueName: \"kubernetes.io/projected/1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3-kube-api-access-2tt72\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:28:34.429725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.429701 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-875bcff54-bhskn"] Nov 01 22:28:34.432186 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:28:34.432159 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02318c1e_d93c_451c_88e4_5a09a1502ced.slice/crio-30cb6b3e2d1f6a6c510968cb58ca9750b6e1ab1e8af37151aa0a0613604eeb62 WatchSource:0}: Error finding container 30cb6b3e2d1f6a6c510968cb58ca9750b6e1ab1e8af37151aa0a0613604eeb62: Status 404 returned error can't find the container with id 30cb6b3e2d1f6a6c510968cb58ca9750b6e1ab1e8af37151aa0a0613604eeb62 Nov 01 22:28:34.491877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.491452 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-87hds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 22:28:34.491877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:34.491506 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 22:28:35.129984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:35.129943 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-875bcff54-bhskn" event={"ID":"02318c1e-d93c-451c-88e4-5a09a1502ced","Type":"ContainerStarted","Data":"30cb6b3e2d1f6a6c510968cb58ca9750b6e1ab1e8af37151aa0a0613604eeb62"} Nov 01 22:28:35.129984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:35.129982 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wvfz7" Nov 01 22:28:35.159765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:35.159737 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvfz7"] Nov 01 22:28:35.165182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:35.165154 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvfz7"] Nov 01 22:28:35.469293 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:35.469215 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3" path="/var/lib/kubelet/pods/1dca83a8-9e9d-48a2-b5d1-a67f4cac45d3/volumes" Nov 01 22:28:35.488139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:35.486215 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-87hds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 22:28:35.488139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:35.486278 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 22:28:36.137141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:36.137095 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-875bcff54-bhskn" event={"ID":"02318c1e-d93c-451c-88e4-5a09a1502ced","Type":"ContainerStarted","Data":"d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2"} Nov 01 22:28:36.154750 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:36.154699 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-875bcff54-bhskn" podStartSLOduration=2.056694108 podStartE2EDuration="3.154683773s" totalImagesPullingTime="1.097989665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:36.153383944 +0000 UTC m=+328.801883182" watchObservedRunningTime="2026-11-01 22:28:36.154683773 +0000 UTC m=+328.803183009" Nov 01 22:28:36.485617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:36.485522 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-87hds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 22:28:36.485871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:36.485839 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 22:28:37.495138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:37.493466 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-87hds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 22:28:37.495138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:37.493526 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 22:28:38.477496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:38.477465 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:28:38.477784 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:38.477762 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:28:38.478361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:28:38.478341 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 22:28:45.585970 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:28:45.585932 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:29:15.585834 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:29:15.585795 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:29:45.585435 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:29:45.585400 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:29:55.644591 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.644558 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg"] Nov 01 22:29:55.648012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.647987 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.665275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.665250 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wtxww\"" Nov 01 22:29:55.665416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.665357 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 01 22:29:55.667187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.667157 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg"] Nov 01 22:29:55.675827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.675801 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 22:29:55.676003 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.675839 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 22:29:55.800941 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.800901 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.801189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.800974 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68fff20-1ed3-4f68-a012-444c11020805-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.801189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.801054 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.801189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.801083 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.801189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.801108 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5g2\" (UniqueName: \"kubernetes.io/projected/e68fff20-1ed3-4f68-a012-444c11020805-kube-api-access-zt5g2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.801434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.801230 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.902565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.902469 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.902565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.902540 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68fff20-1ed3-4f68-a012-444c11020805-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.902806 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.902573 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.902806 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.902600 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.902806 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.902633 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5g2\" (UniqueName: \"kubernetes.io/projected/e68fff20-1ed3-4f68-a012-444c11020805-kube-api-access-zt5g2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.902806 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.902685 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.903051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.902911 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.903051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.902930 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.903051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.903034 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.906538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.906505 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68fff20-1ed3-4f68-a012-444c11020805-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.906754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.906733 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68fff20-1ed3-4f68-a012-444c11020805-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.922775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.922724 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5g2\" (UniqueName: \"kubernetes.io/projected/e68fff20-1ed3-4f68-a012-444c11020805-kube-api-access-zt5g2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg\" (UID: \"e68fff20-1ed3-4f68-a012-444c11020805\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:55.959787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:55.959755 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:56.259634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.259559 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg"] Nov 01 22:29:56.261881 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:29:56.261852 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode68fff20_1ed3_4f68_a012_444c11020805.slice/crio-55e7b6d7a7cd2e244ec971ce9a13f8745c70892eb9b7539f891c6aa7be35cbe7 WatchSource:0}: Error finding container 55e7b6d7a7cd2e244ec971ce9a13f8745c70892eb9b7539f891c6aa7be35cbe7: Status 404 returned error can't find the container with id 55e7b6d7a7cd2e244ec971ce9a13f8745c70892eb9b7539f891c6aa7be35cbe7 Nov 01 22:29:56.496175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.496140 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" event={"ID":"e68fff20-1ed3-4f68-a012-444c11020805","Type":"ContainerStarted","Data":"55e7b6d7a7cd2e244ec971ce9a13f8745c70892eb9b7539f891c6aa7be35cbe7"} Nov 01 22:29:56.784784 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.783505 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg"] Nov 01 22:29:56.787681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.786998 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:56.794594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.794141 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 22:29:56.803558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.803455 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg"] Nov 01 22:29:56.911331 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.911270 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7025b36-aa82-40c1-9fc1-e1e4470498c5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:56.911331 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.911324 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj6mc\" (UniqueName: \"kubernetes.io/projected/a7025b36-aa82-40c1-9fc1-e1e4470498c5-kube-api-access-kj6mc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:56.911549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.911391 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:56.911549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.911440 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:56.911549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.911468 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:56.911549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:56.911511 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.012786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.012749 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.012959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.012817 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.012959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.012842 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.012959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.012880 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.012959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.012943 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7025b36-aa82-40c1-9fc1-e1e4470498c5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.013151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.012978 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj6mc\" (UniqueName: \"kubernetes.io/projected/a7025b36-aa82-40c1-9fc1-e1e4470498c5-kube-api-access-kj6mc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.013763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.013736 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.014009 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.013991 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.014890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.014533 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.016415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.016389 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7025b36-aa82-40c1-9fc1-e1e4470498c5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.022515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.022486 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7025b36-aa82-40c1-9fc1-e1e4470498c5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.024607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.024576 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj6mc\" (UniqueName: \"kubernetes.io/projected/a7025b36-aa82-40c1-9fc1-e1e4470498c5-kube-api-access-kj6mc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vvqpg\" (UID: \"a7025b36-aa82-40c1-9fc1-e1e4470498c5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.104741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.104653 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:29:57.390465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.389493 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg"] Nov 01 22:29:57.400772 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:29:57.400728 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7025b36_aa82_40c1_9fc1_e1e4470498c5.slice/crio-721d2a0beef0ef32b96face3899a60c6d31a0cf9b9bd7324c8acbfb434a7877c WatchSource:0}: Error finding container 721d2a0beef0ef32b96face3899a60c6d31a0cf9b9bd7324c8acbfb434a7877c: Status 404 returned error can't find the container with id 721d2a0beef0ef32b96face3899a60c6d31a0cf9b9bd7324c8acbfb434a7877c Nov 01 22:29:57.503459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:57.503405 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" event={"ID":"a7025b36-aa82-40c1-9fc1-e1e4470498c5","Type":"ContainerStarted","Data":"721d2a0beef0ef32b96face3899a60c6d31a0cf9b9bd7324c8acbfb434a7877c"} Nov 01 22:29:58.513474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:58.513427 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" event={"ID":"e68fff20-1ed3-4f68-a012-444c11020805","Type":"ContainerStarted","Data":"4c89620bca5decfa2a1c7e7af10d20c72de5aae44901c4297c2b6016aea60e3e"} Nov 01 22:29:58.515748 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:58.515715 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" event={"ID":"a7025b36-aa82-40c1-9fc1-e1e4470498c5","Type":"ContainerStarted","Data":"dd6435b83bdc8c5ecad627e01f2ba3ee52bb659c8032521c4b0e1caf3e787447"} Nov 01 22:29:58.535024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:58.534966 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podStartSLOduration=2.132415333 podStartE2EDuration="3.534947863s" totalImagesPullingTime="1.40253253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:58.533509151 +0000 UTC m=+411.182008386" watchObservedRunningTime="2026-11-01 22:29:58.534947863 +0000 UTC m=+411.183447099" Nov 01 22:29:58.552803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:58.552753 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" podStartSLOduration=2.288191202 podStartE2EDuration="2.552741128s" totalImagesPullingTime="264.549926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:58.55245055 +0000 UTC m=+411.200949785" watchObservedRunningTime="2026-11-01 22:29:58.552741128 +0000 UTC m=+411.201240394" Nov 01 22:29:58.960532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:58.960448 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:29:59.105709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:29:59.105678 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:30:00.138422 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.138383 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd"] Nov 01 22:30:00.142067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.142047 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:00.148451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.148410 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:30:00.148568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.148549 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:30:00.148643 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.148631 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:30:00.150989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.150951 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd"] Nov 01 22:30:00.248301 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.248267 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5xv\" (UniqueName: \"kubernetes.io/projected/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-kube-api-access-4r5xv\") pod \"maas-api-key-cleanup-29892870-qtzkd\" (UID: \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:00.248457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.248408 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892870-qtzkd\" (UID: \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:00.349616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.349576 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5xv\" (UniqueName: \"kubernetes.io/projected/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-kube-api-access-4r5xv\") pod \"maas-api-key-cleanup-29892870-qtzkd\" (UID: \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:00.349837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.349631 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892870-qtzkd\" (UID: \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:00.350255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.350235 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892870-qtzkd\" (UID: \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:00.358718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.358669 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5xv\" (UniqueName: \"kubernetes.io/projected/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-kube-api-access-4r5xv\") pod \"maas-api-key-cleanup-29892870-qtzkd\" (UID: \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:00.456711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.456632 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:00.750958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:00.750927 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd"] Nov 01 22:30:00.754026 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:30:00.753997 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode481b1fe_cf7d_4ffa_b6e9_3de86f99d0e6.slice/crio-a406a70d205de09832d7e7496d3c14c69c6e9505e842300ad224cb843c905a26 WatchSource:0}: Error finding container a406a70d205de09832d7e7496d3c14c69c6e9505e842300ad224cb843c905a26: Status 404 returned error can't find the container with id a406a70d205de09832d7e7496d3c14c69c6e9505e842300ad224cb843c905a26 Nov 01 22:30:01.547567 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:01.547530 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" event={"ID":"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6","Type":"ContainerStarted","Data":"a406a70d205de09832d7e7496d3c14c69c6e9505e842300ad224cb843c905a26"} Nov 01 22:30:04.564488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:04.564447 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" event={"ID":"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6","Type":"ContainerStarted","Data":"e873681266e1ef3597b33d886857580d0390687c9232dcd394e6a4f5b0797301"} Nov 01 22:30:05.569875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:05.569841 6287 generic.go:356] "Generic (PLEG): container finished" podID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerID="e873681266e1ef3597b33d886857580d0390687c9232dcd394e6a4f5b0797301" exitCode=60 Nov 01 22:30:05.570308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:05.569929 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" event={"ID":"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6","Type":"ContainerDied","Data":"e873681266e1ef3597b33d886857580d0390687c9232dcd394e6a4f5b0797301"} Nov 01 22:30:05.570308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:05.570107 6287 scope.go:122] "RemoveContainer" containerID="e873681266e1ef3597b33d886857580d0390687c9232dcd394e6a4f5b0797301" Nov 01 22:30:05.960305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:05.960230 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:30:05.973605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:05.973581 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:30:06.576656 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:06.576625 6287 generic.go:356] "Generic (PLEG): container finished" podID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerID="7544a6325b24fb6fda98a2a99ed21a386e84aa5b157e99f9dc93938217a10100" exitCode=60 Nov 01 22:30:06.577081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:06.576705 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" event={"ID":"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6","Type":"ContainerDied","Data":"7544a6325b24fb6fda98a2a99ed21a386e84aa5b157e99f9dc93938217a10100"} Nov 01 22:30:06.577081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:06.576761 6287 scope.go:122] "RemoveContainer" containerID="e873681266e1ef3597b33d886857580d0390687c9232dcd394e6a4f5b0797301" Nov 01 22:30:06.577081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:06.577006 6287 scope.go:122] "RemoveContainer" containerID="7544a6325b24fb6fda98a2a99ed21a386e84aa5b157e99f9dc93938217a10100" Nov 01 22:30:06.577368 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:30:06.577253 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892870-qtzkd_odh-ai-gateway-infra(e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" Nov 01 22:30:06.588700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:06.588678 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:30:07.105372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:07.105335 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:30:07.117639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:07.117612 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:30:07.583513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:07.583484 6287 scope.go:122] "RemoveContainer" containerID="7544a6325b24fb6fda98a2a99ed21a386e84aa5b157e99f9dc93938217a10100" Nov 01 22:30:07.583980 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:30:07.583736 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892870-qtzkd_odh-ai-gateway-infra(e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" Nov 01 22:30:07.595442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:07.595418 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:30:11.018277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.018244 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-548f455f69-d5qjg"] Nov 01 22:30:11.022937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.022921 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-548f455f69-d5qjg" Nov 01 22:30:11.030909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.030885 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-548f455f69-d5qjg"] Nov 01 22:30:11.055796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.055774 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02d479a1-5e3c-4d34-b486-4d8441c95f3a-tls-cert\") pod \"authorino-548f455f69-d5qjg\" (UID: \"02d479a1-5e3c-4d34-b486-4d8441c95f3a\") " pod="kuadrant-system/authorino-548f455f69-d5qjg" Nov 01 22:30:11.055918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.055826 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2425j\" (UniqueName: \"kubernetes.io/projected/02d479a1-5e3c-4d34-b486-4d8441c95f3a-kube-api-access-2425j\") pod \"authorino-548f455f69-d5qjg\" (UID: \"02d479a1-5e3c-4d34-b486-4d8441c95f3a\") " pod="kuadrant-system/authorino-548f455f69-d5qjg" Nov 01 22:30:11.157147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.157087 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02d479a1-5e3c-4d34-b486-4d8441c95f3a-tls-cert\") pod \"authorino-548f455f69-d5qjg\" (UID: \"02d479a1-5e3c-4d34-b486-4d8441c95f3a\") " pod="kuadrant-system/authorino-548f455f69-d5qjg" Nov 01 22:30:11.157274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.157212 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2425j\" (UniqueName: \"kubernetes.io/projected/02d479a1-5e3c-4d34-b486-4d8441c95f3a-kube-api-access-2425j\") pod \"authorino-548f455f69-d5qjg\" (UID: \"02d479a1-5e3c-4d34-b486-4d8441c95f3a\") " pod="kuadrant-system/authorino-548f455f69-d5qjg" Nov 01 22:30:11.159506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.159486 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02d479a1-5e3c-4d34-b486-4d8441c95f3a-tls-cert\") pod \"authorino-548f455f69-d5qjg\" (UID: \"02d479a1-5e3c-4d34-b486-4d8441c95f3a\") " pod="kuadrant-system/authorino-548f455f69-d5qjg" Nov 01 22:30:11.173982 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.173957 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2425j\" (UniqueName: \"kubernetes.io/projected/02d479a1-5e3c-4d34-b486-4d8441c95f3a-kube-api-access-2425j\") pod \"authorino-548f455f69-d5qjg\" (UID: \"02d479a1-5e3c-4d34-b486-4d8441c95f3a\") " pod="kuadrant-system/authorino-548f455f69-d5qjg" Nov 01 22:30:11.335644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.335578 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-548f455f69-d5qjg" Nov 01 22:30:11.605881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:11.605803 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-548f455f69-d5qjg"] Nov 01 22:30:11.609294 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:30:11.609266 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02d479a1_5e3c_4d34_b486_4d8441c95f3a.slice/crio-842336319fb0c586141a06bc3c8160fa9d6b805d5a682fab3dfd8cea6df0d6cb WatchSource:0}: Error finding container 842336319fb0c586141a06bc3c8160fa9d6b805d5a682fab3dfd8cea6df0d6cb: Status 404 returned error can't find the container with id 842336319fb0c586141a06bc3c8160fa9d6b805d5a682fab3dfd8cea6df0d6cb Nov 01 22:30:12.609443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.609361 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-548f455f69-d5qjg" event={"ID":"02d479a1-5e3c-4d34-b486-4d8441c95f3a","Type":"ContainerStarted","Data":"76972f76784829a7a1e6d82efb64d5c78571555a08d20b7dd9aea55dd7014db4"} Nov 01 22:30:12.609443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.609407 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-548f455f69-d5qjg" event={"ID":"02d479a1-5e3c-4d34-b486-4d8441c95f3a","Type":"ContainerStarted","Data":"842336319fb0c586141a06bc3c8160fa9d6b805d5a682fab3dfd8cea6df0d6cb"} Nov 01 22:30:12.627355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.627310 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-548f455f69-d5qjg" podStartSLOduration=1.9019222359999999 podStartE2EDuration="2.627298011s" totalImagesPullingTime="725.375775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:12.626554341 +0000 UTC m=+425.275053575" watchObservedRunningTime="2026-11-01 22:30:12.627298011 +0000 UTC m=+425.275797245" Nov 01 22:30:12.655455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.655397 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-875bcff54-bhskn"] Nov 01 22:30:12.655643 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.655587 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-875bcff54-bhskn" podUID="02318c1e-d93c-451c-88e4-5a09a1502ced" containerName="authorino" containerID="cri-o://d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2" gracePeriod=30 Nov 01 22:30:12.911054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.911029 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:30:12.975396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.975365 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02318c1e-d93c-451c-88e4-5a09a1502ced-kube-api-access-kvm65\" (UniqueName: \"kubernetes.io/projected/02318c1e-d93c-451c-88e4-5a09a1502ced-kube-api-access-kvm65\") pod \"02318c1e-d93c-451c-88e4-5a09a1502ced\" (UID: \"02318c1e-d93c-451c-88e4-5a09a1502ced\") " Nov 01 22:30:12.975588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.975469 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02318c1e-d93c-451c-88e4-5a09a1502ced-tls-cert\" (UniqueName: \"kubernetes.io/secret/02318c1e-d93c-451c-88e4-5a09a1502ced-tls-cert\") pod \"02318c1e-d93c-451c-88e4-5a09a1502ced\" (UID: \"02318c1e-d93c-451c-88e4-5a09a1502ced\") " Nov 01 22:30:12.977430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.977400 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02318c1e-d93c-451c-88e4-5a09a1502ced-kube-api-access-kvm65" pod "02318c1e-d93c-451c-88e4-5a09a1502ced" (UID: "02318c1e-d93c-451c-88e4-5a09a1502ced"). InnerVolumeSpecName "kube-api-access-kvm65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:12.985176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:12.985148 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02318c1e-d93c-451c-88e4-5a09a1502ced-tls-cert" pod "02318c1e-d93c-451c-88e4-5a09a1502ced" (UID: "02318c1e-d93c-451c-88e4-5a09a1502ced"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:13.076815 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.076776 6287 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02318c1e-d93c-451c-88e4-5a09a1502ced-tls-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:13.076815 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.076809 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvm65\" (UniqueName: \"kubernetes.io/projected/02318c1e-d93c-451c-88e4-5a09a1502ced-kube-api-access-kvm65\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:13.188441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.188368 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf"] Nov 01 22:30:13.188789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.188765 6287 cpu_manager.go:414] "containerMap: removing container" podUID="02318c1e-d93c-451c-88e4-5a09a1502ced" containerName="authorino" Nov 01 22:30:13.188789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.188784 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02318c1e-d93c-451c-88e4-5a09a1502ced" containerName="authorino" Nov 01 22:30:13.188944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.188858 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="02318c1e-d93c-451c-88e4-5a09a1502ced" containerName="authorino" Nov 01 22:30:13.191888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.191871 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:13.202786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.200040 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 22:30:13.202786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.201401 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-n48lj\"" Nov 01 22:30:13.202786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.202314 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 22:30:13.202786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.202388 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 22:30:13.202786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.202570 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 22:30:13.217585 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.217550 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf"] Nov 01 22:30:13.278773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.278733 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b222g\" (UniqueName: \"kubernetes.io/projected/46eaa4f2-0b89-4356-bc94-c73726f6aba1-kube-api-access-b222g\") pod \"ai-gateway-controller-5f85cd9cfc-jkzdf\" (UID: \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:13.279011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.278787 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46eaa4f2-0b89-4356-bc94-c73726f6aba1-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-jkzdf\" (UID: \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:13.380261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.380227 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46eaa4f2-0b89-4356-bc94-c73726f6aba1-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-jkzdf\" (UID: \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:13.380432 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.380304 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b222g\" (UniqueName: \"kubernetes.io/projected/46eaa4f2-0b89-4356-bc94-c73726f6aba1-kube-api-access-b222g\") pod \"ai-gateway-controller-5f85cd9cfc-jkzdf\" (UID: \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:13.382998 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.382969 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46eaa4f2-0b89-4356-bc94-c73726f6aba1-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-jkzdf\" (UID: \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:13.389803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.389771 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b222g\" (UniqueName: \"kubernetes.io/projected/46eaa4f2-0b89-4356-bc94-c73726f6aba1-kube-api-access-b222g\") pod \"ai-gateway-controller-5f85cd9cfc-jkzdf\" (UID: \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:13.523230 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.523167 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:13.621148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.620487 6287 generic.go:356] "Generic (PLEG): container finished" podID="02318c1e-d93c-451c-88e4-5a09a1502ced" containerID="d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2" exitCode=0 Nov 01 22:30:13.621148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.621129 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-875bcff54-bhskn" Nov 01 22:30:13.621642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.621169 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-875bcff54-bhskn" event={"ID":"02318c1e-d93c-451c-88e4-5a09a1502ced","Type":"ContainerDied","Data":"d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2"} Nov 01 22:30:13.621642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.621232 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-875bcff54-bhskn" event={"ID":"02318c1e-d93c-451c-88e4-5a09a1502ced","Type":"ContainerDied","Data":"30cb6b3e2d1f6a6c510968cb58ca9750b6e1ab1e8af37151aa0a0613604eeb62"} Nov 01 22:30:13.621642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.621258 6287 scope.go:122] "RemoveContainer" containerID="d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2" Nov 01 22:30:13.631778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.631760 6287 scope.go:122] "RemoveContainer" containerID="d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2" Nov 01 22:30:13.632170 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:30:13.632140 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2\": container with ID starting with d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2 not found: ID does not exist" containerID="d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2" Nov 01 22:30:13.632244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.632184 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2"} err="failed to get container status \"d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2\": rpc error: code = NotFound desc = could not find container \"d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2\": container with ID starting with d4e0a196f09993c645b9c5cd543930d7920383ac09db25d73383473554fcc6f2 not found: ID does not exist" Nov 01 22:30:13.643586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.643546 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-875bcff54-bhskn"] Nov 01 22:30:13.648034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.648009 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-875bcff54-bhskn"] Nov 01 22:30:13.826309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:13.826286 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf"] Nov 01 22:30:13.829177 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:30:13.829142 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46eaa4f2_0b89_4356_bc94_c73726f6aba1.slice/crio-81b341c146cae33db2d2f33f9d75182085eab4501c7c0b003d363512f0cf546a WatchSource:0}: Error finding container 81b341c146cae33db2d2f33f9d75182085eab4501c7c0b003d363512f0cf546a: Status 404 returned error can't find the container with id 81b341c146cae33db2d2f33f9d75182085eab4501c7c0b003d363512f0cf546a Nov 01 22:30:14.627426 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:14.627390 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" event={"ID":"46eaa4f2-0b89-4356-bc94-c73726f6aba1","Type":"ContainerStarted","Data":"81b341c146cae33db2d2f33f9d75182085eab4501c7c0b003d363512f0cf546a"} Nov 01 22:30:15.471250 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:15.471211 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02318c1e-d93c-451c-88e4-5a09a1502ced" path="/var/lib/kubelet/pods/02318c1e-d93c-451c-88e4-5a09a1502ced/volumes" Nov 01 22:30:15.600357 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:30:15.600298 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:30:18.463810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:18.463771 6287 scope.go:122] "RemoveContainer" containerID="7544a6325b24fb6fda98a2a99ed21a386e84aa5b157e99f9dc93938217a10100" Nov 01 22:30:18.665765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:18.665729 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" event={"ID":"46eaa4f2-0b89-4356-bc94-c73726f6aba1","Type":"ContainerStarted","Data":"577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45"} Nov 01 22:30:18.665933 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:18.665861 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:18.682850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:18.682793 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" podStartSLOduration=1.7206381899999998 podStartE2EDuration="5.682779646s" totalImagesPullingTime="3.962141456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:18.681534907 +0000 UTC m=+431.330034144" watchObservedRunningTime="2026-11-01 22:30:18.682779646 +0000 UTC m=+431.331278940" Nov 01 22:30:19.673208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.673179 6287 generic.go:356] "Generic (PLEG): container finished" podID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerID="99ca799990684be7735528f0930a0b2f4bf7ee84da052f351bc9b3523e507b3c" exitCode=60 Nov 01 22:30:19.673758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.673256 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" event={"ID":"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6","Type":"ContainerDied","Data":"99ca799990684be7735528f0930a0b2f4bf7ee84da052f351bc9b3523e507b3c"} Nov 01 22:30:19.673758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.673324 6287 scope.go:122] "RemoveContainer" containerID="7544a6325b24fb6fda98a2a99ed21a386e84aa5b157e99f9dc93938217a10100" Nov 01 22:30:19.673758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.673743 6287 scope.go:122] "RemoveContainer" containerID="99ca799990684be7735528f0930a0b2f4bf7ee84da052f351bc9b3523e507b3c" Nov 01 22:30:19.674007 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:30:19.673979 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892870-qtzkd_odh-ai-gateway-infra(e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" Nov 01 22:30:19.720531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.720501 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rx8f2"] Nov 01 22:30:19.724811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.724788 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:19.735271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.735244 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rx8f2"] Nov 01 22:30:19.759188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.759163 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rx8f2\" (UID: \"faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:19.759363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.759344 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wd6q\" (UniqueName: \"kubernetes.io/projected/faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3-kube-api-access-5wd6q\") pod \"payload-processing-55fbbbf9dc-rx8f2\" (UID: \"faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:19.804478 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.804446 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 22:30:19.804647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.804446 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nnlzm\"" Nov 01 22:30:19.860094 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.860052 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wd6q\" (UniqueName: \"kubernetes.io/projected/faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3-kube-api-access-5wd6q\") pod \"payload-processing-55fbbbf9dc-rx8f2\" (UID: \"faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:19.860275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.860124 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rx8f2\" (UID: \"faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:19.860836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.860816 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rx8f2\" (UID: \"faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:19.867944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:19.867917 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wd6q\" (UniqueName: \"kubernetes.io/projected/faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3-kube-api-access-5wd6q\") pod \"payload-processing-55fbbbf9dc-rx8f2\" (UID: \"faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:20.038009 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:20.037975 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:20.312594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:20.312561 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rx8f2"] Nov 01 22:30:20.314965 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:30:20.314929 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf889a0_6a6a_4ecd_98b1_fd2b2d550ce3.slice/crio-6da5d13aab1364b2b5efe87997820fc7e44574795435eac970190f6839c008ed WatchSource:0}: Error finding container 6da5d13aab1364b2b5efe87997820fc7e44574795435eac970190f6839c008ed: Status 404 returned error can't find the container with id 6da5d13aab1364b2b5efe87997820fc7e44574795435eac970190f6839c008ed Nov 01 22:30:20.679753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:20.679664 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" event={"ID":"faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3","Type":"ContainerStarted","Data":"6da5d13aab1364b2b5efe87997820fc7e44574795435eac970190f6839c008ed"} Nov 01 22:30:20.713271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:20.713231 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd"] Nov 01 22:30:20.836392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:20.836365 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:20.979999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:20.979917 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-kube-api-access-4r5xv\" (UniqueName: \"kubernetes.io/projected/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-kube-api-access-4r5xv\") pod \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\" (UID: \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\") " Nov 01 22:30:20.979999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:20.979979 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-maas-api-service-ca\") pod \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\" (UID: \"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6\") " Nov 01 22:30:20.980412 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:20.980382 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-maas-api-service-ca" pod "e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" (UID: "e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:20.983252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:20.983178 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-kube-api-access-4r5xv" pod "e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" (UID: "e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6"). InnerVolumeSpecName "kube-api-access-4r5xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:21.081504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:21.081468 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5xv\" (UniqueName: \"kubernetes.io/projected/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-kube-api-access-4r5xv\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:21.081504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:21.081506 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:21.689408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:21.689375 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" Nov 01 22:30:21.689408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:21.689396 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd" event={"ID":"e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6","Type":"ContainerDied","Data":"a406a70d205de09832d7e7496d3c14c69c6e9505e842300ad224cb843c905a26"} Nov 01 22:30:21.689948 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:21.689461 6287 scope.go:122] "RemoveContainer" containerID="99ca799990684be7735528f0930a0b2f4bf7ee84da052f351bc9b3523e507b3c" Nov 01 22:30:21.706698 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:21.706670 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd"] Nov 01 22:30:21.711067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:21.711043 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-qtzkd"] Nov 01 22:30:22.515039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.515011 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm"] Nov 01 22:30:22.515432 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.515415 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerName="cleanup" Nov 01 22:30:22.515491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.515433 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerName="cleanup" Nov 01 22:30:22.515491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.515459 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerName="cleanup" Nov 01 22:30:22.515491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.515464 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerName="cleanup" Nov 01 22:30:22.515596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.515527 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerName="cleanup" Nov 01 22:30:22.515596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.515537 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerName="cleanup" Nov 01 22:30:22.515596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.515543 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerName="cleanup" Nov 01 22:30:22.518687 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.518666 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.538140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.538055 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm"] Nov 01 22:30:22.595141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.595081 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.595294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.595235 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.595441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.595423 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.595492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.595454 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxlv\" (UniqueName: \"kubernetes.io/projected/e65c3aec-3fe4-4c4f-8689-cbabab25e810-kube-api-access-fzxlv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.595492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.595483 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.595589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.595571 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.595668 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.595654 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.595703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.595686 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.595801 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.595785 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696180 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" event={"ID":"faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3","Type":"ContainerStarted","Data":"dc16a434c8694ee1f8a0477178b3675ab6d90b91a8bc36927cddb458dc98cc8a"} Nov 01 22:30:22.696275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696203 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696248 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:22.696275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696251 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696357 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696392 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696453 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696477 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696498 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxlv\" (UniqueName: \"kubernetes.io/projected/e65c3aec-3fe4-4c4f-8689-cbabab25e810-kube-api-access-fzxlv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696533 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696559 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696682 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696722 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.696918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.696797 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.697483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.697074 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.697483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.697176 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.699018 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.698991 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.699234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.699215 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.705207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.705185 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxlv\" (UniqueName: \"kubernetes.io/projected/e65c3aec-3fe4-4c4f-8689-cbabab25e810-kube-api-access-fzxlv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.706646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.706617 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65c3aec-3fe4-4c4f-8689-cbabab25e810-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8m4jm\" (UID: \"e65c3aec-3fe4-4c4f-8689-cbabab25e810\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:22.714766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.714727 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" podStartSLOduration=1.5854212570000001 podStartE2EDuration="3.714714432s" totalImagesPullingTime="2.129293175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:22.713671617 +0000 UTC m=+435.362170853" watchObservedRunningTime="2026-11-01 22:30:22.714714432 +0000 UTC m=+435.363213666" Nov 01 22:30:22.832354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:22.832325 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:23.111933 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:23.111908 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm"] Nov 01 22:30:23.113542 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:30:23.113517 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65c3aec_3fe4_4c4f_8689_cbabab25e810.slice/crio-1fea05c586aa03820a990e25b2347132230d185c57c59f9f2b178a20f29bc6b7 WatchSource:0}: Error finding container 1fea05c586aa03820a990e25b2347132230d185c57c59f9f2b178a20f29bc6b7: Status 404 returned error can't find the container with id 1fea05c586aa03820a990e25b2347132230d185c57c59f9f2b178a20f29bc6b7 Nov 01 22:30:23.115751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:23.115723 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:30:23.115812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:23.115785 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:30:23.115846 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:23.115818 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:30:23.469573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:23.469494 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" path="/var/lib/kubelet/pods/e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6/volumes" Nov 01 22:30:23.704440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:23.704401 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" event={"ID":"e65c3aec-3fe4-4c4f-8689-cbabab25e810","Type":"ContainerStarted","Data":"41849290eef1f4dca49f231e44a8e0347cfa17b2029b69bf056000915a36b53f"} Nov 01 22:30:23.704851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:23.704467 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" event={"ID":"e65c3aec-3fe4-4c4f-8689-cbabab25e810","Type":"ContainerStarted","Data":"1fea05c586aa03820a990e25b2347132230d185c57c59f9f2b178a20f29bc6b7"} Nov 01 22:30:23.726702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:23.726600 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" podStartSLOduration=1.726584211 podStartE2EDuration="1.726584211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:23.72525373 +0000 UTC m=+436.373752962" watchObservedRunningTime="2026-11-01 22:30:23.726584211 +0000 UTC m=+436.375083447" Nov 01 22:30:23.832732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:23.832703 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:24.835089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:24.835048 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8m4jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:30:24.835518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:24.835141 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" podUID="e65c3aec-3fe4-4c4f-8689-cbabab25e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:30:25.832822 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:25.832789 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8m4jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:30:25.832999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:25.832850 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" podUID="e65c3aec-3fe4-4c4f-8689-cbabab25e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:30:26.833167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:26.833103 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8m4jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:30:26.833559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:26.833183 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" podUID="e65c3aec-3fe4-4c4f-8689-cbabab25e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:30:27.832950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:27.832913 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8m4jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:30:27.833157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:27.832973 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" podUID="e65c3aec-3fe4-4c4f-8689-cbabab25e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:30:28.832952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:28.832916 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8m4jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:30:28.833356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:28.832973 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" podUID="e65c3aec-3fe4-4c4f-8689-cbabab25e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:30:29.676153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:29.676108 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:29.833526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:29.833489 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8m4jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:30:29.834071 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:29.833539 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" podUID="e65c3aec-3fe4-4c4f-8689-cbabab25e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:30:29.940965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:29.940880 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6b6d674b9-llr7t"] Nov 01 22:30:29.941497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:29.941475 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerName="cleanup" Nov 01 22:30:29.941560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:29.941503 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e481b1fe-cf7d-4ffa-b6e9-3de86f99d0e6" containerName="cleanup" Nov 01 22:30:29.947980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:29.947956 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:29.953934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:29.953903 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6b6d674b9-llr7t"] Nov 01 22:30:30.068712 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:30.068676 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dh5w\" (UniqueName: \"kubernetes.io/projected/4459f450-7dab-4975-b2ac-02b042d1e004-kube-api-access-6dh5w\") pod \"ai-gateway-controller-6b6d674b9-llr7t\" (UID: \"4459f450-7dab-4975-b2ac-02b042d1e004\") " pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:30.068894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:30.068747 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4459f450-7dab-4975-b2ac-02b042d1e004-metrics-certs\") pod \"ai-gateway-controller-6b6d674b9-llr7t\" (UID: \"4459f450-7dab-4975-b2ac-02b042d1e004\") " pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:30.170401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:30.170361 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4459f450-7dab-4975-b2ac-02b042d1e004-metrics-certs\") pod \"ai-gateway-controller-6b6d674b9-llr7t\" (UID: \"4459f450-7dab-4975-b2ac-02b042d1e004\") " pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:30.170557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:30.170486 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dh5w\" (UniqueName: \"kubernetes.io/projected/4459f450-7dab-4975-b2ac-02b042d1e004-kube-api-access-6dh5w\") pod \"ai-gateway-controller-6b6d674b9-llr7t\" (UID: \"4459f450-7dab-4975-b2ac-02b042d1e004\") " pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:30.172968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:30.172936 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4459f450-7dab-4975-b2ac-02b042d1e004-metrics-certs\") pod \"ai-gateway-controller-6b6d674b9-llr7t\" (UID: \"4459f450-7dab-4975-b2ac-02b042d1e004\") " pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:30.179706 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:30.179680 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dh5w\" (UniqueName: \"kubernetes.io/projected/4459f450-7dab-4975-b2ac-02b042d1e004-kube-api-access-6dh5w\") pod \"ai-gateway-controller-6b6d674b9-llr7t\" (UID: \"4459f450-7dab-4975-b2ac-02b042d1e004\") " pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:30.264327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:30.264288 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:30.541674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:30.541642 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6b6d674b9-llr7t"] Nov 01 22:30:30.542411 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:30:30.542381 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4459f450_7dab_4975_b2ac_02b042d1e004.slice/crio-ef9a2551fc708840e6e078e382aaf28ad98cbb261ba85e27eb1f5f94d8ea2128 WatchSource:0}: Error finding container ef9a2551fc708840e6e078e382aaf28ad98cbb261ba85e27eb1f5f94d8ea2128: Status 404 returned error can't find the container with id ef9a2551fc708840e6e078e382aaf28ad98cbb261ba85e27eb1f5f94d8ea2128 Nov 01 22:30:30.740480 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:30.740440 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" event={"ID":"4459f450-7dab-4975-b2ac-02b042d1e004","Type":"ContainerStarted","Data":"ef9a2551fc708840e6e078e382aaf28ad98cbb261ba85e27eb1f5f94d8ea2128"} Nov 01 22:30:31.323546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:31.323511 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:31.324060 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:31.323971 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:31.324777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:31.324757 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" Nov 01 22:30:31.747343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:31.747305 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" event={"ID":"4459f450-7dab-4975-b2ac-02b042d1e004","Type":"ContainerStarted","Data":"a59572c575dcaed869ede8666ae2d549d35c6f70e161ebe869e35f7b86b6f926"} Nov 01 22:30:31.747575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:31.747551 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:31.766252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:31.766206 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podStartSLOduration=2.180498733 podStartE2EDuration="2.76619346s" totalImagesPullingTime="585.694727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:31.765137561 +0000 UTC m=+444.413636794" watchObservedRunningTime="2026-11-01 22:30:31.76619346 +0000 UTC m=+444.414692695" Nov 01 22:30:33.707351 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:33.707324 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" Nov 01 22:30:42.754215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:42.754179 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:30:42.814926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:42.814893 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf"] Nov 01 22:30:42.815208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:42.815182 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" podUID="46eaa4f2-0b89-4356-bc94-c73726f6aba1" containerName="manager" containerID="cri-o://577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45" gracePeriod=30 Nov 01 22:30:43.062189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.062165 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:43.188320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.188278 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46eaa4f2-0b89-4356-bc94-c73726f6aba1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46eaa4f2-0b89-4356-bc94-c73726f6aba1-metrics-certs\") pod \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\" (UID: \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\") " Nov 01 22:30:43.188473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.188350 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46eaa4f2-0b89-4356-bc94-c73726f6aba1-kube-api-access-b222g\" (UniqueName: \"kubernetes.io/projected/46eaa4f2-0b89-4356-bc94-c73726f6aba1-kube-api-access-b222g\") pod \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\" (UID: \"46eaa4f2-0b89-4356-bc94-c73726f6aba1\") " Nov 01 22:30:43.190897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.190813 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46eaa4f2-0b89-4356-bc94-c73726f6aba1-metrics-certs" pod "46eaa4f2-0b89-4356-bc94-c73726f6aba1" (UID: "46eaa4f2-0b89-4356-bc94-c73726f6aba1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:30:43.191027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.190930 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46eaa4f2-0b89-4356-bc94-c73726f6aba1-kube-api-access-b222g" pod "46eaa4f2-0b89-4356-bc94-c73726f6aba1" (UID: "46eaa4f2-0b89-4356-bc94-c73726f6aba1"). InnerVolumeSpecName "kube-api-access-b222g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:43.289615 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.289573 6287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46eaa4f2-0b89-4356-bc94-c73726f6aba1-metrics-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:43.289615 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.289613 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b222g\" (UniqueName: \"kubernetes.io/projected/46eaa4f2-0b89-4356-bc94-c73726f6aba1-kube-api-access-b222g\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:43.807040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.807005 6287 generic.go:356] "Generic (PLEG): container finished" podID="46eaa4f2-0b89-4356-bc94-c73726f6aba1" containerID="577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45" exitCode=0 Nov 01 22:30:43.807496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.807054 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" event={"ID":"46eaa4f2-0b89-4356-bc94-c73726f6aba1","Type":"ContainerDied","Data":"577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45"} Nov 01 22:30:43.807496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.807092 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" event={"ID":"46eaa4f2-0b89-4356-bc94-c73726f6aba1","Type":"ContainerDied","Data":"81b341c146cae33db2d2f33f9d75182085eab4501c7c0b003d363512f0cf546a"} Nov 01 22:30:43.807496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.807054 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf" Nov 01 22:30:43.807496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.807142 6287 scope.go:122] "RemoveContainer" containerID="577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45" Nov 01 22:30:43.815922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.815793 6287 scope.go:122] "RemoveContainer" containerID="577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45" Nov 01 22:30:43.816108 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:30:43.816087 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45\": container with ID starting with 577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45 not found: ID does not exist" containerID="577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45" Nov 01 22:30:43.816184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.816133 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45"} err="failed to get container status \"577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45\": rpc error: code = NotFound desc = could not find container \"577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45\": container with ID starting with 577dd0be72a9eb0484688aca02c30e1196206b6c4b95d7f3bed88637463ffb45 not found: ID does not exist" Nov 01 22:30:43.838202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.838178 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf"] Nov 01 22:30:43.846191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:43.846172 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-jkzdf"] Nov 01 22:30:45.469803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:30:45.469770 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46eaa4f2-0b89-4356-bc94-c73726f6aba1" path="/var/lib/kubelet/pods/46eaa4f2-0b89-4356-bc94-c73726f6aba1/volumes" Nov 01 22:30:45.596674 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:30:45.596635 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:31:13.660621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.660584 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6"] Nov 01 22:31:13.661045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.660979 6287 cpu_manager.go:414] "containerMap: removing container" podUID="46eaa4f2-0b89-4356-bc94-c73726f6aba1" containerName="manager" Nov 01 22:31:13.661045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.660992 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46eaa4f2-0b89-4356-bc94-c73726f6aba1" containerName="manager" Nov 01 22:31:13.661153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.661048 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="46eaa4f2-0b89-4356-bc94-c73726f6aba1" containerName="manager" Nov 01 22:31:13.669545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.669522 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.676517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.676484 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6"] Nov 01 22:31:13.689469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.689437 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-00e49d-openshift-default-dockercfg-xhjdm\"" Nov 01 22:31:13.742281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.742246 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.742281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.742282 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.742511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.742339 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.742511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.742389 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.742511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.742452 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.742511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.742491 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.742702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.742593 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667699ea-ed8e-4055-ae52-3bd87e5e966d-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.742702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.742621 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pxdf\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-kube-api-access-2pxdf\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.742813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.742731 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.832039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.832004 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b"] Nov 01 22:31:13.835541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.835516 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.844005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.843982 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844023 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844048 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844223 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844271 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844339 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844381 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844410 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667699ea-ed8e-4055-ae52-3bd87e5e966d-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844428 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844436 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxdf\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-kube-api-access-2pxdf\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844456 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.844826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.844780 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.845469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.845426 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.846572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.846539 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667699ea-ed8e-4055-ae52-3bd87e5e966d-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.847528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.847506 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.847619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.847603 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b"] Nov 01 22:31:13.849224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.849204 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.858304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.858282 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxdf\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-kube-api-access-2pxdf\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.858387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.858301 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.863039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.863021 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-aa441b-openshift-default-dockercfg-kjt95\"" Nov 01 22:31:13.945141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.945047 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68acbcb-66c9-4def-af31-184d7544248d-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.945141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.945086 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.945141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.945126 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.945347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.945159 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrqh\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-kube-api-access-5wrqh\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.945347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.945317 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.945414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.945370 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68acbcb-66c9-4def-af31-184d7544248d-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.945461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.945445 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.945522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.945489 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.945596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.945580 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:13.984263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.984235 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:13.999379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:13.999353 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv"] Nov 01 22:31:14.008487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.008460 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.017898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.017836 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv"] Nov 01 22:31:14.034817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.034789 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e25182-openshift-default-dockercfg-pf4mc\"" Nov 01 22:31:14.046200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.046169 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.046323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.046223 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.046323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.046249 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68acbcb-66c9-4def-af31-184d7544248d-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.046323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.046283 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdq2g\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-kube-api-access-vdq2g\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.046311 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.046781 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrqh\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-kube-api-access-5wrqh\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.046852 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.046884 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.046913 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.047343 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.047396 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.047437 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.047481 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.047523 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68acbcb-66c9-4def-af31-184d7544248d-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.047569 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479b82c5-9b9a-4abe-9372-38c5305bb753-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.047594 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.047635 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.047680 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.048443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.048080 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.049710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.049540 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.051078 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.049881 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.051078 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.050781 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68acbcb-66c9-4def-af31-184d7544248d-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.051078 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.051005 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.051078 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.051032 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.053069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.053036 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68acbcb-66c9-4def-af31-184d7544248d-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.064209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.064184 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.068921 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.068893 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrqh\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-kube-api-access-5wrqh\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.148588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148552 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.148766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148602 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479b82c5-9b9a-4abe-9372-38c5305bb753-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.148766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148621 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.148766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148640 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.148766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148682 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdq2g\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-kube-api-access-vdq2g\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.148766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148705 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.149025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148829 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.149025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148874 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:14.149025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148888 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.149025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.148927 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.149246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.149100 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.149246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.149205 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.149387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.149363 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.149470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.149451 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.149523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.149509 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479b82c5-9b9a-4abe-9372-38c5305bb753-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.151009 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.150986 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.151429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.151412 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.158219 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.158199 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.158395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.158378 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdq2g\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-kube-api-access-vdq2g\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.279314 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.279287 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6"] Nov 01 22:31:14.281745 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:14.281717 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667699ea_ed8e_4055_ae52_3bd87e5e966d.slice/crio-ef7b718aa95dee62590ed0ac09a1b5da935680051b502f513a6095e79dc3f25b WatchSource:0}: Error finding container ef7b718aa95dee62590ed0ac09a1b5da935680051b502f513a6095e79dc3f25b: Status 404 returned error can't find the container with id ef7b718aa95dee62590ed0ac09a1b5da935680051b502f513a6095e79dc3f25b Nov 01 22:31:14.284328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.284283 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:14.284434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.284377 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:14.284434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.284420 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:14.323759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.323731 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:14.443558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.443534 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b"] Nov 01 22:31:14.446387 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:14.446329 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68acbcb_66c9_4def_af31_184d7544248d.slice/crio-70ffe4be9f5e03fc99a5e259445e73176a030d6144ae1a729f1205a4d2190143 WatchSource:0}: Error finding container 70ffe4be9f5e03fc99a5e259445e73176a030d6144ae1a729f1205a4d2190143: Status 404 returned error can't find the container with id 70ffe4be9f5e03fc99a5e259445e73176a030d6144ae1a729f1205a4d2190143 Nov 01 22:31:14.449876 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.449838 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:14.450000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.449976 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:14.450068 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.450031 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:14.619277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.619252 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv"] Nov 01 22:31:14.622077 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:14.622042 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479b82c5_9b9a_4abe_9372_38c5305bb753.slice/crio-e8039b1eed9c600d7652b8c8b829489d23ec318afd15659df099646b02d720ce WatchSource:0}: Error finding container e8039b1eed9c600d7652b8c8b829489d23ec318afd15659df099646b02d720ce: Status 404 returned error can't find the container with id e8039b1eed9c600d7652b8c8b829489d23ec318afd15659df099646b02d720ce Nov 01 22:31:14.624838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.624799 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:14.624923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.624869 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:14.624923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.624903 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:14.966005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.965966 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" event={"ID":"c68acbcb-66c9-4def-af31-184d7544248d","Type":"ContainerStarted","Data":"4c9554c4500e6f2bf3967e64df2314825d4f45c1388bb1fec9737e2c3e42adc0"} Nov 01 22:31:14.966442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.966030 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" event={"ID":"c68acbcb-66c9-4def-af31-184d7544248d","Type":"ContainerStarted","Data":"70ffe4be9f5e03fc99a5e259445e73176a030d6144ae1a729f1205a4d2190143"} Nov 01 22:31:14.967801 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.967771 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" event={"ID":"667699ea-ed8e-4055-ae52-3bd87e5e966d","Type":"ContainerStarted","Data":"4d1d7b9a5725a3dafd8d44e347fb3a037c3e7a6c7797747054919431fc2982e7"} Nov 01 22:31:14.967932 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.967823 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" event={"ID":"667699ea-ed8e-4055-ae52-3bd87e5e966d","Type":"ContainerStarted","Data":"ef7b718aa95dee62590ed0ac09a1b5da935680051b502f513a6095e79dc3f25b"} Nov 01 22:31:14.969494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.969473 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" event={"ID":"479b82c5-9b9a-4abe-9372-38c5305bb753","Type":"ContainerStarted","Data":"56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585"} Nov 01 22:31:14.969597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.969513 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" event={"ID":"479b82c5-9b9a-4abe-9372-38c5305bb753","Type":"ContainerStarted","Data":"e8039b1eed9c600d7652b8c8b829489d23ec318afd15659df099646b02d720ce"} Nov 01 22:31:14.984466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.984443 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:14.987133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:14.987071 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" podStartSLOduration=1.987058188 podStartE2EDuration="1.987058188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:14.985466993 +0000 UTC m=+487.633966240" watchObservedRunningTime="2026-11-01 22:31:14.987058188 +0000 UTC m=+487.635557424" Nov 01 22:31:15.005628 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.005513 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podStartSLOduration=2.005496379 podStartE2EDuration="2.005496379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:15.005149349 +0000 UTC m=+487.653648584" watchObservedRunningTime="2026-11-01 22:31:15.005496379 +0000 UTC m=+487.653995616" Nov 01 22:31:15.025227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.025188 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" podStartSLOduration=2.025176583 podStartE2EDuration="2.025176583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:15.024215077 +0000 UTC m=+487.672714313" watchObservedRunningTime="2026-11-01 22:31:15.025176583 +0000 UTC m=+487.673675818" Nov 01 22:31:15.150136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.150072 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:15.156644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.156615 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:15.325153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.325094 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:15.330250 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.330228 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:15.587184 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:31:15.587073 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:31:15.975575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.975490 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:15.976006 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.975850 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:15.976336 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.976316 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:15.976424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.976373 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:15.989867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:15.989843 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:16.980947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:16.980913 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:16.981621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:16.981602 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:18.204859 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.204820 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns"] Nov 01 22:31:18.208674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.208651 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:18.216361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.216289 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns"] Nov 01 22:31:18.218398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.218376 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b42cc4\"" Nov 01 22:31:18.238534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.238468 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw"] Nov 01 22:31:18.242302 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.242280 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:18.251619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.250798 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b42cc4-dockercfg-972gb\"" Nov 01 22:31:18.252820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.252791 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw"] Nov 01 22:31:18.291566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.291525 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxlf\" (UniqueName: \"kubernetes.io/projected/8ac2dd5a-11ff-408d-96f0-dac764508392-kube-api-access-zjxlf\") pod \"payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns\" (UID: \"8ac2dd5a-11ff-408d-96f0-dac764508392\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:18.291730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.291664 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9644b916-6603-48dc-b6e8-993bc7cec7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw\" (UID: \"9644b916-6603-48dc-b6e8-993bc7cec7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:18.291819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.291713 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtt7\" (UniqueName: \"kubernetes.io/projected/9644b916-6603-48dc-b6e8-993bc7cec7f8-kube-api-access-7rtt7\") pod \"payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw\" (UID: \"9644b916-6603-48dc-b6e8-993bc7cec7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:18.291946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.291916 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2dd5a-11ff-408d-96f0-dac764508392-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns\" (UID: \"8ac2dd5a-11ff-408d-96f0-dac764508392\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:18.392949 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.392909 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2dd5a-11ff-408d-96f0-dac764508392-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns\" (UID: \"8ac2dd5a-11ff-408d-96f0-dac764508392\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:18.393167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.392965 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxlf\" (UniqueName: \"kubernetes.io/projected/8ac2dd5a-11ff-408d-96f0-dac764508392-kube-api-access-zjxlf\") pod \"payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns\" (UID: \"8ac2dd5a-11ff-408d-96f0-dac764508392\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:18.393167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.393021 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9644b916-6603-48dc-b6e8-993bc7cec7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw\" (UID: \"9644b916-6603-48dc-b6e8-993bc7cec7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:18.393167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.393050 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtt7\" (UniqueName: \"kubernetes.io/projected/9644b916-6603-48dc-b6e8-993bc7cec7f8-kube-api-access-7rtt7\") pod \"payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw\" (UID: \"9644b916-6603-48dc-b6e8-993bc7cec7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:18.393767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.393744 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9644b916-6603-48dc-b6e8-993bc7cec7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw\" (UID: \"9644b916-6603-48dc-b6e8-993bc7cec7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:18.393818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.393751 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2dd5a-11ff-408d-96f0-dac764508392-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns\" (UID: \"8ac2dd5a-11ff-408d-96f0-dac764508392\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:18.404315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.404288 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxlf\" (UniqueName: \"kubernetes.io/projected/8ac2dd5a-11ff-408d-96f0-dac764508392-kube-api-access-zjxlf\") pod \"payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns\" (UID: \"8ac2dd5a-11ff-408d-96f0-dac764508392\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:18.405994 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.405972 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtt7\" (UniqueName: \"kubernetes.io/projected/9644b916-6603-48dc-b6e8-993bc7cec7f8-kube-api-access-7rtt7\") pod \"payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw\" (UID: \"9644b916-6603-48dc-b6e8-993bc7cec7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:18.533415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.533369 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:18.556509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.556482 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:18.831274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.831219 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns"] Nov 01 22:31:18.833691 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:18.833664 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac2dd5a_11ff_408d_96f0_dac764508392.slice/crio-387a813ee4fb626b6433edb8a2d1eb2ca50bc3edd69c3030c27d8b28baffe6fd WatchSource:0}: Error finding container 387a813ee4fb626b6433edb8a2d1eb2ca50bc3edd69c3030c27d8b28baffe6fd: Status 404 returned error can't find the container with id 387a813ee4fb626b6433edb8a2d1eb2ca50bc3edd69c3030c27d8b28baffe6fd Nov 01 22:31:18.838722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.838696 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw"] Nov 01 22:31:18.846438 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:18.846411 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9644b916_6603_48dc_b6e8_993bc7cec7f8.slice/crio-86a4d20f4f7ddc092607f8e3392353302700af067244a51a968c1248e5c93c66 WatchSource:0}: Error finding container 86a4d20f4f7ddc092607f8e3392353302700af067244a51a968c1248e5c93c66: Status 404 returned error can't find the container with id 86a4d20f4f7ddc092607f8e3392353302700af067244a51a968c1248e5c93c66 Nov 01 22:31:18.992367 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.992329 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" event={"ID":"8ac2dd5a-11ff-408d-96f0-dac764508392","Type":"ContainerStarted","Data":"6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785"} Nov 01 22:31:18.992531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.992398 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:18.992531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.992420 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" event={"ID":"8ac2dd5a-11ff-408d-96f0-dac764508392","Type":"ContainerStarted","Data":"387a813ee4fb626b6433edb8a2d1eb2ca50bc3edd69c3030c27d8b28baffe6fd"} Nov 01 22:31:18.994241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.994210 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" event={"ID":"9644b916-6603-48dc-b6e8-993bc7cec7f8","Type":"ContainerStarted","Data":"3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5"} Nov 01 22:31:18.994379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:18.994257 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" event={"ID":"9644b916-6603-48dc-b6e8-993bc7cec7f8","Type":"ContainerStarted","Data":"86a4d20f4f7ddc092607f8e3392353302700af067244a51a968c1248e5c93c66"} Nov 01 22:31:19.015064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.015013 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" podStartSLOduration=1.014999302 podStartE2EDuration="1.014999302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:19.013585622 +0000 UTC m=+491.662084857" watchObservedRunningTime="2026-11-01 22:31:19.014999302 +0000 UTC m=+491.663498537" Nov 01 22:31:19.047100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.047043 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" podStartSLOduration=1.047028966 podStartE2EDuration="1.047028966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:19.044309834 +0000 UTC m=+491.692809070" watchObservedRunningTime="2026-11-01 22:31:19.047028966 +0000 UTC m=+491.695528226" Nov 01 22:31:19.141106 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.141016 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj"] Nov 01 22:31:19.144831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.144810 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:19.150034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.150011 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e25182-dockercfg-9922m\"" Nov 01 22:31:19.150034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.150012 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e25182\"" Nov 01 22:31:19.157645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.157620 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf"] Nov 01 22:31:19.160908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.160888 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj"] Nov 01 22:31:19.161020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.161005 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:19.172681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.172655 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf"] Nov 01 22:31:19.200143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.200100 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55badcc9-0370-42d9-a059-5537ff2acdb6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf\" (UID: \"55badcc9-0370-42d9-a059-5537ff2acdb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:19.200143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.200145 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4xg6\" (UniqueName: \"kubernetes.io/projected/55badcc9-0370-42d9-a059-5537ff2acdb6-kube-api-access-l4xg6\") pod \"payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf\" (UID: \"55badcc9-0370-42d9-a059-5537ff2acdb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:19.200334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.200229 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xcw\" (UniqueName: \"kubernetes.io/projected/85879320-4b31-4f5e-a902-797e6f510711-kube-api-access-j4xcw\") pod \"payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj\" (UID: \"85879320-4b31-4f5e-a902-797e6f510711\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:19.200334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.200250 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85879320-4b31-4f5e-a902-797e6f510711-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj\" (UID: \"85879320-4b31-4f5e-a902-797e6f510711\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:19.301282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.301243 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55badcc9-0370-42d9-a059-5537ff2acdb6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf\" (UID: \"55badcc9-0370-42d9-a059-5537ff2acdb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:19.301767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.301295 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4xg6\" (UniqueName: \"kubernetes.io/projected/55badcc9-0370-42d9-a059-5537ff2acdb6-kube-api-access-l4xg6\") pod \"payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf\" (UID: \"55badcc9-0370-42d9-a059-5537ff2acdb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:19.301767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.301645 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xcw\" (UniqueName: \"kubernetes.io/projected/85879320-4b31-4f5e-a902-797e6f510711-kube-api-access-j4xcw\") pod \"payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj\" (UID: \"85879320-4b31-4f5e-a902-797e6f510711\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:19.301767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.301680 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85879320-4b31-4f5e-a902-797e6f510711-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj\" (UID: \"85879320-4b31-4f5e-a902-797e6f510711\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:19.302902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.302881 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85879320-4b31-4f5e-a902-797e6f510711-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj\" (UID: \"85879320-4b31-4f5e-a902-797e6f510711\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:19.303029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.302994 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55badcc9-0370-42d9-a059-5537ff2acdb6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf\" (UID: \"55badcc9-0370-42d9-a059-5537ff2acdb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:19.310636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.310600 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xcw\" (UniqueName: \"kubernetes.io/projected/85879320-4b31-4f5e-a902-797e6f510711-kube-api-access-j4xcw\") pod \"payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj\" (UID: \"85879320-4b31-4f5e-a902-797e6f510711\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:19.310763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.310609 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4xg6\" (UniqueName: \"kubernetes.io/projected/55badcc9-0370-42d9-a059-5537ff2acdb6-kube-api-access-l4xg6\") pod \"payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf\" (UID: \"55badcc9-0370-42d9-a059-5537ff2acdb6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:19.460350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.460267 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:19.477527 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.477479 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:19.703854 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.703817 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78"] Nov 01 22:31:19.708715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.708693 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc"] Nov 01 22:31:19.708832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.708818 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:19.712093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.712043 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:19.718868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.718842 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78"] Nov 01 22:31:19.721680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.721657 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-00e49d\"" Nov 01 22:31:19.726951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.726930 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc"] Nov 01 22:31:19.748614 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.748591 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj"] Nov 01 22:31:19.749729 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:19.749706 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85879320_4b31_4f5e_a902_797e6f510711.slice/crio-2dfabce55f72d205deb0851f57ef80ee32c07e6063b6fa2c2f9d9da0cd43251c WatchSource:0}: Error finding container 2dfabce55f72d205deb0851f57ef80ee32c07e6063b6fa2c2f9d9da0cd43251c: Status 404 returned error can't find the container with id 2dfabce55f72d205deb0851f57ef80ee32c07e6063b6fa2c2f9d9da0cd43251c Nov 01 22:31:19.772995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.772973 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf"] Nov 01 22:31:19.778057 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:19.778032 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55badcc9_0370_42d9_a059_5537ff2acdb6.slice/crio-51a91c1fee85c6de96929307c757455a199a1f60e520a46c28a7c31d8418f310 WatchSource:0}: Error finding container 51a91c1fee85c6de96929307c757455a199a1f60e520a46c28a7c31d8418f310: Status 404 returned error can't find the container with id 51a91c1fee85c6de96929307c757455a199a1f60e520a46c28a7c31d8418f310 Nov 01 22:31:19.807231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.807201 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2z59\" (UniqueName: \"kubernetes.io/projected/c46a3da2-5f01-4bba-a31a-a3adc458d938-kube-api-access-m2z59\") pod \"payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78\" (UID: \"c46a3da2-5f01-4bba-a31a-a3adc458d938\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:19.807335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.807288 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhbv\" (UniqueName: \"kubernetes.io/projected/5d9fc308-bd50-44de-b033-3e9cb02b11c7-kube-api-access-vfhbv\") pod \"payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc\" (UID: \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:19.807405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.807376 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fc308-bd50-44de-b033-3e9cb02b11c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc\" (UID: \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:19.807532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.807515 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46a3da2-5f01-4bba-a31a-a3adc458d938-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78\" (UID: \"c46a3da2-5f01-4bba-a31a-a3adc458d938\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:19.908839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.908805 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2z59\" (UniqueName: \"kubernetes.io/projected/c46a3da2-5f01-4bba-a31a-a3adc458d938-kube-api-access-m2z59\") pod \"payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78\" (UID: \"c46a3da2-5f01-4bba-a31a-a3adc458d938\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:19.909039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.908890 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhbv\" (UniqueName: \"kubernetes.io/projected/5d9fc308-bd50-44de-b033-3e9cb02b11c7-kube-api-access-vfhbv\") pod \"payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc\" (UID: \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:19.909039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.908929 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fc308-bd50-44de-b033-3e9cb02b11c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc\" (UID: \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:19.909039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.908991 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46a3da2-5f01-4bba-a31a-a3adc458d938-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78\" (UID: \"c46a3da2-5f01-4bba-a31a-a3adc458d938\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:19.910938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.910911 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46a3da2-5f01-4bba-a31a-a3adc458d938-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78\" (UID: \"c46a3da2-5f01-4bba-a31a-a3adc458d938\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:19.911473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.911451 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fc308-bd50-44de-b033-3e9cb02b11c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc\" (UID: \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:19.917674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.917627 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2z59\" (UniqueName: \"kubernetes.io/projected/c46a3da2-5f01-4bba-a31a-a3adc458d938-kube-api-access-m2z59\") pod \"payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78\" (UID: \"c46a3da2-5f01-4bba-a31a-a3adc458d938\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:19.917830 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:19.917804 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhbv\" (UniqueName: \"kubernetes.io/projected/5d9fc308-bd50-44de-b033-3e9cb02b11c7-kube-api-access-vfhbv\") pod \"payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc\" (UID: \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:20.002603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.002566 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" event={"ID":"55badcc9-0370-42d9-a059-5537ff2acdb6","Type":"ContainerStarted","Data":"0098098fbfa47ed552b9089cb3f4a5b331a9742b0f0416795644b5bf74b3c2c0"} Nov 01 22:31:20.002783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.002657 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" event={"ID":"55badcc9-0370-42d9-a059-5537ff2acdb6","Type":"ContainerStarted","Data":"51a91c1fee85c6de96929307c757455a199a1f60e520a46c28a7c31d8418f310"} Nov 01 22:31:20.002783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.002758 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:20.004471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.004442 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" event={"ID":"85879320-4b31-4f5e-a902-797e6f510711","Type":"ContainerStarted","Data":"ece38054aef0b497bca4917b4b5c7169e1875f75c49cc825e88d1ef2669e5265"} Nov 01 22:31:20.004600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.004492 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" event={"ID":"85879320-4b31-4f5e-a902-797e6f510711","Type":"ContainerStarted","Data":"2dfabce55f72d205deb0851f57ef80ee32c07e6063b6fa2c2f9d9da0cd43251c"} Nov 01 22:31:20.004690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.004677 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:20.004846 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.004824 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:20.027437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.027412 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:20.041483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.041456 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:20.056447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.056398 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" podStartSLOduration=1.056383559 podStartE2EDuration="1.056383559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:20.053225297 +0000 UTC m=+492.701724533" watchObservedRunningTime="2026-11-01 22:31:20.056383559 +0000 UTC m=+492.704882849" Nov 01 22:31:20.108024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.107978 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" podStartSLOduration=1.107964494 podStartE2EDuration="1.107964494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:20.10447485 +0000 UTC m=+492.752974108" watchObservedRunningTime="2026-11-01 22:31:20.107964494 +0000 UTC m=+492.756463728" Nov 01 22:31:20.208677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.208643 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t"] Nov 01 22:31:20.213931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.213911 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4"] Nov 01 22:31:20.214038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.214021 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:20.217512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.217490 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:20.223438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.223416 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-aa441b\"" Nov 01 22:31:20.223529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.223454 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-aa441b-dockercfg-dcgkm\"" Nov 01 22:31:20.226333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.226306 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t"] Nov 01 22:31:20.229040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.229016 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4"] Nov 01 22:31:20.314435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.314334 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgn6\" (UniqueName: \"kubernetes.io/projected/230255df-b6dc-43d5-bcd7-364d2396ed06-kube-api-access-gdgn6\") pod \"payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4\" (UID: \"230255df-b6dc-43d5-bcd7-364d2396ed06\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:20.317855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.317715 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230255df-b6dc-43d5-bcd7-364d2396ed06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4\" (UID: \"230255df-b6dc-43d5-bcd7-364d2396ed06\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:20.317855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.317828 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v799r\" (UniqueName: \"kubernetes.io/projected/da7f0a84-0c8c-441e-8851-828d7d9949e8-kube-api-access-v799r\") pod \"payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t\" (UID: \"da7f0a84-0c8c-441e-8851-828d7d9949e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:20.318162 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.317880 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7f0a84-0c8c-441e-8851-828d7d9949e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t\" (UID: \"da7f0a84-0c8c-441e-8851-828d7d9949e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:20.322366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.322337 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78"] Nov 01 22:31:20.327988 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:20.327960 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46a3da2_5f01_4bba_a31a_a3adc458d938.slice/crio-f3e2873948ebe35e1d16f166aef9d1d19cdd44e0d064217ed5e1bc912fdeb7dc WatchSource:0}: Error finding container f3e2873948ebe35e1d16f166aef9d1d19cdd44e0d064217ed5e1bc912fdeb7dc: Status 404 returned error can't find the container with id f3e2873948ebe35e1d16f166aef9d1d19cdd44e0d064217ed5e1bc912fdeb7dc Nov 01 22:31:20.334042 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.334018 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc"] Nov 01 22:31:20.344268 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:20.344228 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9fc308_bd50_44de_b033_3e9cb02b11c7.slice/crio-cd1e6ba218f69c6c6f6f693d6292faa01ac73271eb9781a8e6cdcfe53a643f7f WatchSource:0}: Error finding container cd1e6ba218f69c6c6f6f693d6292faa01ac73271eb9781a8e6cdcfe53a643f7f: Status 404 returned error can't find the container with id cd1e6ba218f69c6c6f6f693d6292faa01ac73271eb9781a8e6cdcfe53a643f7f Nov 01 22:31:20.418722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.418694 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v799r\" (UniqueName: \"kubernetes.io/projected/da7f0a84-0c8c-441e-8851-828d7d9949e8-kube-api-access-v799r\") pod \"payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t\" (UID: \"da7f0a84-0c8c-441e-8851-828d7d9949e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:20.418858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.418737 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7f0a84-0c8c-441e-8851-828d7d9949e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t\" (UID: \"da7f0a84-0c8c-441e-8851-828d7d9949e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:20.418858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.418820 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgn6\" (UniqueName: \"kubernetes.io/projected/230255df-b6dc-43d5-bcd7-364d2396ed06-kube-api-access-gdgn6\") pod \"payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4\" (UID: \"230255df-b6dc-43d5-bcd7-364d2396ed06\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:20.419073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.418879 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230255df-b6dc-43d5-bcd7-364d2396ed06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4\" (UID: \"230255df-b6dc-43d5-bcd7-364d2396ed06\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:20.419732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.419681 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7f0a84-0c8c-441e-8851-828d7d9949e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t\" (UID: \"da7f0a84-0c8c-441e-8851-828d7d9949e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:20.419732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.419706 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230255df-b6dc-43d5-bcd7-364d2396ed06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4\" (UID: \"230255df-b6dc-43d5-bcd7-364d2396ed06\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:20.427438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.427409 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v799r\" (UniqueName: \"kubernetes.io/projected/da7f0a84-0c8c-441e-8851-828d7d9949e8-kube-api-access-v799r\") pod \"payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t\" (UID: \"da7f0a84-0c8c-441e-8851-828d7d9949e8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:20.427656 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.427637 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgn6\" (UniqueName: \"kubernetes.io/projected/230255df-b6dc-43d5-bcd7-364d2396ed06-kube-api-access-gdgn6\") pod \"payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4\" (UID: \"230255df-b6dc-43d5-bcd7-364d2396ed06\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:20.531298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.531263 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:20.538360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.538337 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:20.815502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.815473 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t"] Nov 01 22:31:20.816787 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:20.816750 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7f0a84_0c8c_441e_8851_828d7d9949e8.slice/crio-fe232bbe78e91f2cb376f5c4aa452fad8e5d7bcf9f928aca4fdeaff89373ec10 WatchSource:0}: Error finding container fe232bbe78e91f2cb376f5c4aa452fad8e5d7bcf9f928aca4fdeaff89373ec10: Status 404 returned error can't find the container with id fe232bbe78e91f2cb376f5c4aa452fad8e5d7bcf9f928aca4fdeaff89373ec10 Nov 01 22:31:20.822593 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.822556 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4"] Nov 01 22:31:20.824509 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:20.824480 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230255df_b6dc_43d5_bcd7_364d2396ed06.slice/crio-6918b00a88d16cee7f2d8eaf2ba97701f150e0781afd2a9d63972e574a5f7a7d WatchSource:0}: Error finding container 6918b00a88d16cee7f2d8eaf2ba97701f150e0781afd2a9d63972e574a5f7a7d: Status 404 returned error can't find the container with id 6918b00a88d16cee7f2d8eaf2ba97701f150e0781afd2a9d63972e574a5f7a7d Nov 01 22:31:20.871164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.871139 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs"] Nov 01 22:31:20.876513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.876125 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:20.887818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.887793 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs"] Nov 01 22:31:20.892232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.892191 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-381659\"" Nov 01 22:31:20.925923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.925885 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5wc\" (UniqueName: \"kubernetes.io/projected/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-kube-api-access-dp5wc\") pod \"payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs\" (UID: \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:20.926085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:20.926021 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs\" (UID: \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:21.016790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.016746 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" event={"ID":"c46a3da2-5f01-4bba-a31a-a3adc458d938","Type":"ContainerStarted","Data":"c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179"} Nov 01 22:31:21.016964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.016819 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" event={"ID":"c46a3da2-5f01-4bba-a31a-a3adc458d938","Type":"ContainerStarted","Data":"f3e2873948ebe35e1d16f166aef9d1d19cdd44e0d064217ed5e1bc912fdeb7dc"} Nov 01 22:31:21.017709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.017680 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:21.019858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.019826 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" event={"ID":"230255df-b6dc-43d5-bcd7-364d2396ed06","Type":"ContainerStarted","Data":"8d7714b871d4fd2a2ce06f9634f4c4499dd9f673db04a774c83a19eb1ef3dda1"} Nov 01 22:31:21.020043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.020024 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" event={"ID":"230255df-b6dc-43d5-bcd7-364d2396ed06","Type":"ContainerStarted","Data":"6918b00a88d16cee7f2d8eaf2ba97701f150e0781afd2a9d63972e574a5f7a7d"} Nov 01 22:31:21.020600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.020548 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:21.023226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.023198 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" event={"ID":"da7f0a84-0c8c-441e-8851-828d7d9949e8","Type":"ContainerStarted","Data":"50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415"} Nov 01 22:31:21.023333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.023244 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" event={"ID":"da7f0a84-0c8c-441e-8851-828d7d9949e8","Type":"ContainerStarted","Data":"fe232bbe78e91f2cb376f5c4aa452fad8e5d7bcf9f928aca4fdeaff89373ec10"} Nov 01 22:31:21.023776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.023753 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:21.027375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.027328 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5wc\" (UniqueName: \"kubernetes.io/projected/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-kube-api-access-dp5wc\") pod \"payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs\" (UID: \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:21.027793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.027671 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs\" (UID: \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:21.029711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.029667 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs\" (UID: \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:21.039667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.039628 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" event={"ID":"5d9fc308-bd50-44de-b033-3e9cb02b11c7","Type":"ContainerStarted","Data":"4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69"} Nov 01 22:31:21.039830 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.039688 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" event={"ID":"5d9fc308-bd50-44de-b033-3e9cb02b11c7","Type":"ContainerStarted","Data":"cd1e6ba218f69c6c6f6f693d6292faa01ac73271eb9781a8e6cdcfe53a643f7f"} Nov 01 22:31:21.039830 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.039715 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:21.049538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.049459 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" podStartSLOduration=2.049437982 podStartE2EDuration="2.049437982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:21.046965461 +0000 UTC m=+493.695464696" watchObservedRunningTime="2026-11-01 22:31:21.049437982 +0000 UTC m=+493.697937239" Nov 01 22:31:21.056418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.056389 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5wc\" (UniqueName: \"kubernetes.io/projected/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-kube-api-access-dp5wc\") pod \"payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs\" (UID: \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:21.067366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.067309 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" podStartSLOduration=2.06729194 podStartE2EDuration="2.06729194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:21.064273866 +0000 UTC m=+493.712773101" watchObservedRunningTime="2026-11-01 22:31:21.06729194 +0000 UTC m=+493.715791176" Nov 01 22:31:21.080874 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.080833 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" podStartSLOduration=1.080816639 podStartE2EDuration="1.080816639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:21.080293244 +0000 UTC m=+493.728792479" watchObservedRunningTime="2026-11-01 22:31:21.080816639 +0000 UTC m=+493.729315926" Nov 01 22:31:21.105626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.105541 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" podStartSLOduration=1.105528911 podStartE2EDuration="1.105528911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:21.102616921 +0000 UTC m=+493.751116157" watchObservedRunningTime="2026-11-01 22:31:21.105528911 +0000 UTC m=+493.754028185" Nov 01 22:31:21.191518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.191493 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:21.480365 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.480333 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs"] Nov 01 22:31:21.481690 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:21.481648 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58e1b66_6f12_4a7b_93d4_02c43e499f7a.slice/crio-f658bb8bd89064f2882bdfc21945568ee488a628acec66bf923e76e532094df3 WatchSource:0}: Error finding container f658bb8bd89064f2882bdfc21945568ee488a628acec66bf923e76e532094df3: Status 404 returned error can't find the container with id f658bb8bd89064f2882bdfc21945568ee488a628acec66bf923e76e532094df3 Nov 01 22:31:21.507355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.507322 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt"] Nov 01 22:31:21.512095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.512077 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:21.528945 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.528922 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-3d8490\"" Nov 01 22:31:21.532063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.532039 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt"] Nov 01 22:31:21.540867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.540849 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn"] Nov 01 22:31:21.545011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.544993 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:21.556176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.556154 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-3d8490-dockercfg-s2f6z\"" Nov 01 22:31:21.557530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.557506 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn"] Nov 01 22:31:21.635260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.635228 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddlg\" (UniqueName: \"kubernetes.io/projected/b00ff221-8630-4a3a-b9c3-37e90a1394fa-kube-api-access-dddlg\") pod \"payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt\" (UID: \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:21.635400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.635286 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27fls\" (UniqueName: \"kubernetes.io/projected/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-kube-api-access-27fls\") pod \"payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn\" (UID: \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:21.635400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.635312 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00ff221-8630-4a3a-b9c3-37e90a1394fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt\" (UID: \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:21.635487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.635461 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn\" (UID: \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:21.736123 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.736024 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddlg\" (UniqueName: \"kubernetes.io/projected/b00ff221-8630-4a3a-b9c3-37e90a1394fa-kube-api-access-dddlg\") pod \"payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt\" (UID: \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:21.736123 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.736079 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27fls\" (UniqueName: \"kubernetes.io/projected/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-kube-api-access-27fls\") pod \"payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn\" (UID: \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:21.736343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.736153 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00ff221-8630-4a3a-b9c3-37e90a1394fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt\" (UID: \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:21.736343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.736225 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn\" (UID: \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:21.736976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.736953 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn\" (UID: \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:21.737138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.737089 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00ff221-8630-4a3a-b9c3-37e90a1394fa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt\" (UID: \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:21.748912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.748885 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27fls\" (UniqueName: \"kubernetes.io/projected/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-kube-api-access-27fls\") pod \"payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn\" (UID: \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:21.749526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.749503 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddlg\" (UniqueName: \"kubernetes.io/projected/b00ff221-8630-4a3a-b9c3-37e90a1394fa-kube-api-access-dddlg\") pod \"payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt\" (UID: \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:21.833556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.833516 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:21.861708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:21.861676 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:22.051564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:22.051529 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" event={"ID":"f58e1b66-6f12-4a7b-93d4-02c43e499f7a","Type":"ContainerStarted","Data":"8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd"} Nov 01 22:31:22.051843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:22.051809 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" event={"ID":"f58e1b66-6f12-4a7b-93d4-02c43e499f7a","Type":"ContainerStarted","Data":"f658bb8bd89064f2882bdfc21945568ee488a628acec66bf923e76e532094df3"} Nov 01 22:31:22.052408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:22.052353 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:22.070597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:22.070540 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" podStartSLOduration=2.070522167 podStartE2EDuration="2.070522167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:22.067638086 +0000 UTC m=+494.716137321" watchObservedRunningTime="2026-11-01 22:31:22.070522167 +0000 UTC m=+494.719021402" Nov 01 22:31:22.118813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:22.118774 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt"] Nov 01 22:31:22.118943 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:22.118783 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00ff221_8630_4a3a_b9c3_37e90a1394fa.slice/crio-5883d6b14f2da8c6b8a60bb5b153eea3916513c9b57e9d2a46835a6dbc954458 WatchSource:0}: Error finding container 5883d6b14f2da8c6b8a60bb5b153eea3916513c9b57e9d2a46835a6dbc954458: Status 404 returned error can't find the container with id 5883d6b14f2da8c6b8a60bb5b153eea3916513c9b57e9d2a46835a6dbc954458 Nov 01 22:31:22.151389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:22.151365 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn"] Nov 01 22:31:22.153689 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:22.153661 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3f18bd_9b22_49cc_b6b9_5ad5ed6a014a.slice/crio-41a434a9efd06d5c485a4440b427a45e6c6ce934d969380f89f18ad317dacec5 WatchSource:0}: Error finding container 41a434a9efd06d5c485a4440b427a45e6c6ce934d969380f89f18ad317dacec5: Status 404 returned error can't find the container with id 41a434a9efd06d5c485a4440b427a45e6c6ce934d969380f89f18ad317dacec5 Nov 01 22:31:23.058695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:23.058654 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" event={"ID":"b00ff221-8630-4a3a-b9c3-37e90a1394fa","Type":"ContainerStarted","Data":"33a300b560ac797a0c7c18a2c6dbe9e46ed51f480f13f301adfb08e88d537c97"} Nov 01 22:31:23.059167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:23.058752 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" event={"ID":"b00ff221-8630-4a3a-b9c3-37e90a1394fa","Type":"ContainerStarted","Data":"5883d6b14f2da8c6b8a60bb5b153eea3916513c9b57e9d2a46835a6dbc954458"} Nov 01 22:31:23.059167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:23.058853 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:23.060237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:23.060206 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" event={"ID":"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a","Type":"ContainerStarted","Data":"124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748"} Nov 01 22:31:23.060328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:23.060257 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" event={"ID":"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a","Type":"ContainerStarted","Data":"41a434a9efd06d5c485a4440b427a45e6c6ce934d969380f89f18ad317dacec5"} Nov 01 22:31:23.075889 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:23.075844 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" podStartSLOduration=2.0758317059999998 podStartE2EDuration="2.075831706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:23.073733178 +0000 UTC m=+495.722232410" watchObservedRunningTime="2026-11-01 22:31:23.075831706 +0000 UTC m=+495.724330943" Nov 01 22:31:23.091850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:23.091804 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" podStartSLOduration=2.091788706 podStartE2EDuration="2.091788706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:23.089583735 +0000 UTC m=+495.738082972" watchObservedRunningTime="2026-11-01 22:31:23.091788706 +0000 UTC m=+495.740287940" Nov 01 22:31:24.065619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:24.065587 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:30.007401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:30.007366 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:31:31.043226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:31.042436 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:31:31.043735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:31.043627 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:31:31.044211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:31.044188 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:31:32.034294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:32.034253 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:31:32.034489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:32.034314 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:31:32.056369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:32.055436 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:31:32.718627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:32.717407 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": read tcp 10.133.0.2:38480->10.133.0.45:15021: read: connection reset by peer" start-of-body= Nov 01 22:31:32.718627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:32.717473 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": read tcp 10.133.0.2:38480->10.133.0.45:15021: read: connection reset by peer" Nov 01 22:31:33.063951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:33.063922 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:31:33.064423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:33.064009 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:31:33.064423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:33.064061 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:31:33.168272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:33.168236 6287 generic.go:356] "Generic (PLEG): container finished" podID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerID="4d1d7b9a5725a3dafd8d44e347fb3a037c3e7a6c7797747054919431fc2982e7" exitCode=0 Nov 01 22:31:33.180378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:33.168308 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" event={"ID":"667699ea-ed8e-4055-ae52-3bd87e5e966d","Type":"ContainerDied","Data":"4d1d7b9a5725a3dafd8d44e347fb3a037c3e7a6c7797747054919431fc2982e7"} Nov 01 22:31:33.180378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:33.168686 6287 scope.go:122] "RemoveContainer" containerID="4d1d7b9a5725a3dafd8d44e347fb3a037c3e7a6c7797747054919431fc2982e7" Nov 01 22:31:33.180378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:33.170188 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:33.180378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:33.170272 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:33.180378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:33.170320 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:34.069039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.069004 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:31:34.069499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.069257 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:31:34.194946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.194911 6287 generic.go:356] "Generic (PLEG): container finished" podID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerID="56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585" exitCode=0 Nov 01 22:31:34.197332 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.197305 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" event={"ID":"479b82c5-9b9a-4abe-9372-38c5305bb753","Type":"ContainerDied","Data":"56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585"} Nov 01 22:31:34.197948 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.197926 6287 scope.go:122] "RemoveContainer" containerID="56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585" Nov 01 22:31:34.200920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.200883 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:34.201141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.201099 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:34.201266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.201252 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:34.211044 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.211013 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" event={"ID":"667699ea-ed8e-4055-ae52-3bd87e5e966d","Type":"ContainerStarted","Data":"c8f0305aa2413a8f06b600611e0f04c0d3a83a7d8eb9e52527d4a6a57b07ba2d"} Nov 01 22:31:34.988551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.988519 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:34.988945 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.988842 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:31:34.989131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:34.989089 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:31:35.104393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:35.103340 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:31:35.230253 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:35.230223 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" event={"ID":"479b82c5-9b9a-4abe-9372-38c5305bb753","Type":"ContainerStarted","Data":"58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc"} Nov 01 22:31:35.325194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:35.324495 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:35.325194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:35.324632 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:31:35.325194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:35.324678 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:31:35.991674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:35.991209 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:31:35.991674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:35.991269 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:31:36.324679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:36.324598 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:31:36.325167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:36.325140 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:31:36.987134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:36.986525 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:31:36.987134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:36.986594 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:31:37.327090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:37.326547 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:31:37.327090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:37.326623 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:31:37.988138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:37.986546 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:31:37.988138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:37.986624 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:31:38.326198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:38.325533 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:31:38.326198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:38.325594 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:31:38.986153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:38.985556 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:31:38.986153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:38.985615 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:31:39.257732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.255690 6287 generic.go:356] "Generic (PLEG): container finished" podID="c68acbcb-66c9-4def-af31-184d7544248d" containerID="4c9554c4500e6f2bf3967e64df2314825d4f45c1388bb1fec9737e2c3e42adc0" exitCode=0 Nov 01 22:31:39.257732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.255761 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" event={"ID":"c68acbcb-66c9-4def-af31-184d7544248d","Type":"ContainerDied","Data":"4c9554c4500e6f2bf3967e64df2314825d4f45c1388bb1fec9737e2c3e42adc0"} Nov 01 22:31:39.257732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.256346 6287 scope.go:122] "RemoveContainer" containerID="4c9554c4500e6f2bf3967e64df2314825d4f45c1388bb1fec9737e2c3e42adc0" Nov 01 22:31:39.257732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.257726 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:39.258398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.257799 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:39.258398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.257834 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:31:39.328904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.324499 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:31:39.328904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.324564 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:31:39.933138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.930012 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs"] Nov 01 22:31:39.933881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.933853 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:39.962008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.961974 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs"] Nov 01 22:31:39.964772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.964743 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-dshm\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:39.965067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.965049 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-tmp-dir\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:39.965281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.965265 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452829c2-d248-41ef-a116-6810384b6951-tls-certs\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:39.965394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.965380 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-home\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:39.965497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.965483 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njs89\" (UniqueName: \"kubernetes.io/projected/452829c2-d248-41ef-a116-6810384b6951-kube-api-access-njs89\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:39.965586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.965574 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-model-cache\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:39.966978 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.966958 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b42cc4\"/\"default-dockercfg-gczbq\"" Nov 01 22:31:39.967255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.967214 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b42cc4\"/\"simulator-w4-b42cc4-kserve-self-signed-certs\"" Nov 01 22:31:39.967743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.967723 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b42cc4\"/\"openshift-service-ca.crt\"" Nov 01 22:31:39.967834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:39.967758 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b42cc4\"/\"kube-root-ca.crt\"" Nov 01 22:31:40.024136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.020496 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:40.027276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.025684 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:40.028843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.028818 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:31:40.067169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.067140 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-tmp-dir\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.067328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.067204 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452829c2-d248-41ef-a116-6810384b6951-tls-certs\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.067328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.067247 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-home\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.067328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.067273 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njs89\" (UniqueName: \"kubernetes.io/projected/452829c2-d248-41ef-a116-6810384b6951-kube-api-access-njs89\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.067328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.067299 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-model-cache\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.067505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.067369 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-dshm\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.067817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.067797 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-home\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.068020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.068002 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-model-cache\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.068198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.068183 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-tmp-dir\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.070215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.070193 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-dshm\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.091134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.079618 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452829c2-d248-41ef-a116-6810384b6951-tls-certs\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.135410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.134775 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njs89\" (UniqueName: \"kubernetes.io/projected/452829c2-d248-41ef-a116-6810384b6951-kube-api-access-njs89\") pod \"simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.245810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.245778 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:40.267022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.266994 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" event={"ID":"c68acbcb-66c9-4def-af31-184d7544248d","Type":"ContainerStarted","Data":"c2dd10aacbab53cde0f36febcbbdbb1ba36a10cc335425dac98c031eecfc7659"} Nov 01 22:31:40.327561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.327519 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:31:40.327762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.327587 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:31:40.605690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:40.605637 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs"] Nov 01 22:31:41.149613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:41.149579 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:41.150025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:41.149838 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:31:41.150025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:41.149890 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:31:41.297151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:41.297101 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" event={"ID":"452829c2-d248-41ef-a116-6810384b6951","Type":"ContainerStarted","Data":"1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88"} Nov 01 22:31:41.297325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:41.297160 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" event={"ID":"452829c2-d248-41ef-a116-6810384b6951","Type":"ContainerStarted","Data":"8752769cf261a3139631897add97db4875acb94d5143ab150a1be9c3ca3a984a"} Nov 01 22:31:41.328503 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:41.328479 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:41.328772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:41.328754 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:41.329379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:41.329360 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:31:41.344937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:41.344891 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podStartSLOduration=2.344878343 podStartE2EDuration="2.344878343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:41.34279484 +0000 UTC m=+513.991294073" watchObservedRunningTime="2026-11-01 22:31:41.344878343 +0000 UTC m=+513.993377577" Nov 01 22:31:42.149751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:42.149710 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:31:42.150294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:42.149768 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:31:42.246505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:42.246475 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:43.150413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:43.150379 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:31:43.150834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:43.150434 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:31:44.150471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:44.150437 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:31:44.150837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:44.150491 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:31:45.150085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:45.150052 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:31:45.150251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:45.150128 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:31:45.589879 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:31:45.589838 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:31:45.975955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:45.975874 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:46.154335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.154304 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:46.331248 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.331215 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:31:46.912157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.912105 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs"] Nov 01 22:31:46.915703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.915681 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:46.920441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.920419 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-aa441b\"/\"default-dockercfg-hvjd5\"" Nov 01 22:31:46.923372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.922301 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-aa441b\"/\"simulator-w5-aa441b-kserve-self-signed-certs\"" Nov 01 22:31:46.923767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.923746 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-aa441b\"/\"kube-root-ca.crt\"" Nov 01 22:31:46.923891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.923839 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-aa441b\"/\"openshift-service-ca.crt\"" Nov 01 22:31:46.925496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.925477 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs"] Nov 01 22:31:46.974735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.974706 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-model-cache\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:46.974857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.974745 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-dshm\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:46.974857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.974769 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-tmp-dir\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:46.974953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.974880 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr5cq\" (UniqueName: \"kubernetes.io/projected/38090de6-418c-45f3-ae0f-e2905865c702-kube-api-access-sr5cq\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:46.974953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.974934 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38090de6-418c-45f3-ae0f-e2905865c702-tls-certs\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:46.975028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:46.975009 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-home\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.077613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.077579 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-home\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.077848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.077647 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-model-cache\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.077848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.077688 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-dshm\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.077848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.077726 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-tmp-dir\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.077848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.077786 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr5cq\" (UniqueName: \"kubernetes.io/projected/38090de6-418c-45f3-ae0f-e2905865c702-kube-api-access-sr5cq\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.078211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.077858 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38090de6-418c-45f3-ae0f-e2905865c702-tls-certs\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.078380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.078358 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-tmp-dir\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.078781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.078756 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-model-cache\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.079057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.079013 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-home\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.082060 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.082014 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-dshm\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.085838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.083152 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38090de6-418c-45f3-ae0f-e2905865c702-tls-certs\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.086794 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.086750 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr5cq\" (UniqueName: \"kubernetes.io/projected/38090de6-418c-45f3-ae0f-e2905865c702-kube-api-access-sr5cq\") pod \"simulator-w5-aa441b-kserve-54b574d455-n7qrs\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.229093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.229016 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:47.510362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:47.510337 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs"] Nov 01 22:31:47.512288 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:47.512263 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38090de6_418c_45f3_ae0f_e2905865c702.slice/crio-ce914506daf9ad6de89ffd54fbfac8f55d8c0b307dd563e43128458aa600a37b WatchSource:0}: Error finding container ce914506daf9ad6de89ffd54fbfac8f55d8c0b307dd563e43128458aa600a37b: Status 404 returned error can't find the container with id ce914506daf9ad6de89ffd54fbfac8f55d8c0b307dd563e43128458aa600a37b Nov 01 22:31:48.346415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:48.346382 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" event={"ID":"38090de6-418c-45f3-ae0f-e2905865c702","Type":"ContainerStarted","Data":"ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714"} Nov 01 22:31:48.346817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:48.346425 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" event={"ID":"38090de6-418c-45f3-ae0f-e2905865c702","Type":"ContainerStarted","Data":"ce914506daf9ad6de89ffd54fbfac8f55d8c0b307dd563e43128458aa600a37b"} Nov 01 22:31:48.367544 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:48.367492 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podStartSLOduration=2.3674760360000002 podStartE2EDuration="2.367476036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:48.364421945 +0000 UTC m=+521.012921176" watchObservedRunningTime="2026-11-01 22:31:48.367476036 +0000 UTC m=+521.015975270" Nov 01 22:31:49.229393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:49.229354 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:49.985916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:49.985878 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6"] Nov 01 22:31:49.990362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:49.990340 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:49.995700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:49.995661 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-381659\"/\"simulator-w1-381659-kserve-self-signed-certs\"" Nov 01 22:31:49.995700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:49.995671 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-381659\"/\"default-dockercfg-x4b79\"" Nov 01 22:31:50.003852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.003829 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6"] Nov 01 22:31:50.031150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.031107 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-381659\"/\"openshift-service-ca.crt\"" Nov 01 22:31:50.031150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.031146 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-381659\"/\"kube-root-ca.crt\"" Nov 01 22:31:50.121149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.121081 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lzh\" (UniqueName: \"kubernetes.io/projected/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-kube-api-access-g4lzh\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.121149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.121150 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-home\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.121380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.121220 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tls-certs\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.121380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.121247 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-model-cache\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.121380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.121284 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tmp-dir\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.121495 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.121450 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-dshm\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.222305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.222264 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lzh\" (UniqueName: \"kubernetes.io/projected/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-kube-api-access-g4lzh\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.222305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.222309 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-home\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.222536 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.222332 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tls-certs\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.222536 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.222352 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-model-cache\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.222536 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.222392 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tmp-dir\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.222536 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.222461 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-dshm\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.222758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.222736 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-model-cache\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.222824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.222761 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-home\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.222893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.222847 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tmp-dir\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.224688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.224666 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-dshm\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.224938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.224922 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tls-certs\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.230607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.230583 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lzh\" (UniqueName: \"kubernetes.io/projected/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-kube-api-access-g4lzh\") pod \"simulator-w1-381659-kserve-65cb6cb7c8-j9jd6\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.246411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.246376 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:50.258861 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.258833 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:50.306362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.306324 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:50.382358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.382328 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:31:50.582894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:50.582870 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6"] Nov 01 22:31:50.584816 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:31:50.584789 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2cab38c_1368_4340_acd0_5a6d5dbd4ab8.slice/crio-0ec0a574d1cd29040e9efe8f1b78f8a4a304a1986a41be19c9d869715faef631 WatchSource:0}: Error finding container 0ec0a574d1cd29040e9efe8f1b78f8a4a304a1986a41be19c9d869715faef631: Status 404 returned error can't find the container with id 0ec0a574d1cd29040e9efe8f1b78f8a4a304a1986a41be19c9d869715faef631 Nov 01 22:31:51.373443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:51.373404 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" event={"ID":"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8","Type":"ContainerStarted","Data":"47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16"} Nov 01 22:31:51.373443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:51.373446 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" event={"ID":"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8","Type":"ContainerStarted","Data":"0ec0a574d1cd29040e9efe8f1b78f8a4a304a1986a41be19c9d869715faef631"} Nov 01 22:31:51.393344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:51.393294 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podStartSLOduration=2.393280826 podStartE2EDuration="2.393280826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:51.390875966 +0000 UTC m=+524.039375201" watchObservedRunningTime="2026-11-01 22:31:51.393280826 +0000 UTC m=+524.041780087" Nov 01 22:31:52.306831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:52.306797 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:31:57.229549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:57.229508 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:57.242881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:57.242856 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:31:57.418953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:31:57.418919 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:32:00.307052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:00.307013 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:32:00.319888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:00.319863 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:32:00.445429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:00.445396 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:32:03.205303 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.205262 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7"] Nov 01 22:32:03.210946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.210921 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.217428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.217401 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b42cc4\"/\"premium-w4-b42cc4-kserve-self-signed-certs\"" Nov 01 22:32:03.217628 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.217606 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7"] Nov 01 22:32:03.370090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.370044 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-home\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.370090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.370092 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42fbbcb1-cd8b-497b-9588-e3015413a83e-tls-certs\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.370371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.370255 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvst6\" (UniqueName: \"kubernetes.io/projected/42fbbcb1-cd8b-497b-9588-e3015413a83e-kube-api-access-fvst6\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.370371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.370302 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-model-cache\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.370371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.370362 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-dshm\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.370514 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.370390 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-tmp-dir\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.471222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.471137 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-model-cache\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.471222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.471189 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-dshm\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.471222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.471215 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-tmp-dir\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.471520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.471277 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-home\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.471520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.471296 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42fbbcb1-cd8b-497b-9588-e3015413a83e-tls-certs\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.471662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.471361 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvst6\" (UniqueName: \"kubernetes.io/projected/42fbbcb1-cd8b-497b-9588-e3015413a83e-kube-api-access-fvst6\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.471727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.471678 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-tmp-dir\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.471788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.471752 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-home\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.471978 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.471960 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-model-cache\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.473656 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.473636 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-dshm\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.474077 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.474056 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42fbbcb1-cd8b-497b-9588-e3015413a83e-tls-certs\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.483765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.483736 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvst6\" (UniqueName: \"kubernetes.io/projected/42fbbcb1-cd8b-497b-9588-e3015413a83e-kube-api-access-fvst6\") pod \"premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.527295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.527261 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:03.817057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:03.817024 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7"] Nov 01 22:32:03.818653 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:32:03.818622 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42fbbcb1_cd8b_497b_9588_e3015413a83e.slice/crio-f24abf491786727f1481bbda6792556b5937ddabc7b30f0045640837d13d7ba5 WatchSource:0}: Error finding container f24abf491786727f1481bbda6792556b5937ddabc7b30f0045640837d13d7ba5: Status 404 returned error can't find the container with id f24abf491786727f1481bbda6792556b5937ddabc7b30f0045640837d13d7ba5 Nov 01 22:32:04.458157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:04.458097 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" event={"ID":"42fbbcb1-cd8b-497b-9588-e3015413a83e","Type":"ContainerStarted","Data":"7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7"} Nov 01 22:32:04.458556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:04.458170 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" event={"ID":"42fbbcb1-cd8b-497b-9588-e3015413a83e","Type":"ContainerStarted","Data":"f24abf491786727f1481bbda6792556b5937ddabc7b30f0045640837d13d7ba5"} Nov 01 22:32:04.476788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:04.476741 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podStartSLOduration=1.47672804 podStartE2EDuration="1.47672804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:04.475320157 +0000 UTC m=+537.123819394" watchObservedRunningTime="2026-11-01 22:32:04.47672804 +0000 UTC m=+537.125227274" Nov 01 22:32:04.528398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:04.528371 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:13.333999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.333960 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx"] Nov 01 22:32:13.339005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.338984 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.347041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.346994 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx"] Nov 01 22:32:13.347208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.347077 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e25182\"/\"openshift-service-ca.crt\"" Nov 01 22:32:13.347208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.347163 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e25182\"/\"kube-root-ca.crt\"" Nov 01 22:32:13.347776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.347754 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e25182\"/\"default-dockercfg-ln5ll\"" Nov 01 22:32:13.347908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.347888 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e25182\"/\"premium-w0-e25182-kserve-self-signed-certs\"" Nov 01 22:32:13.376717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.376689 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tls-certs\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.376859 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.376736 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tmp-dir\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.376859 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.376828 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-dshm\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.376960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.376869 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-model-cache\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.376960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.376948 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-home\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.377040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.376979 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfd2\" (UniqueName: \"kubernetes.io/projected/2fbf50c1-3598-470d-b539-7ac3dfdf0956-kube-api-access-fqfd2\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.477788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.477757 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-home\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.477995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.477811 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfd2\" (UniqueName: \"kubernetes.io/projected/2fbf50c1-3598-470d-b539-7ac3dfdf0956-kube-api-access-fqfd2\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.477995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.477880 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tls-certs\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.477995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.477916 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tmp-dir\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.477995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.477960 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-dshm\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.478256 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.478233 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-home\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.478315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.478265 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tmp-dir\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.478372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.478314 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-model-cache\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.478611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.478585 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-model-cache\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.480855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.480822 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-dshm\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.481145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.481105 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tls-certs\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.485690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.485651 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfd2\" (UniqueName: \"kubernetes.io/projected/2fbf50c1-3598-470d-b539-7ac3dfdf0956-kube-api-access-fqfd2\") pod \"premium-w0-e25182-kserve-694cbbc687-lzxmx\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.527726 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.527699 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:13.540723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.540702 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:13.653351 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.653285 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:13.927702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:13.927650 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx"] Nov 01 22:32:13.928790 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:32:13.928761 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fbf50c1_3598_470d_b539_7ac3dfdf0956.slice/crio-22ee6968132baedcef5ad7c9a90bbf7e3e7542350ec874f2686da876214a6fb0 WatchSource:0}: Error finding container 22ee6968132baedcef5ad7c9a90bbf7e3e7542350ec874f2686da876214a6fb0: Status 404 returned error can't find the container with id 22ee6968132baedcef5ad7c9a90bbf7e3e7542350ec874f2686da876214a6fb0 Nov 01 22:32:14.529496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:14.529463 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" event={"ID":"2fbf50c1-3598-470d-b539-7ac3dfdf0956","Type":"ContainerStarted","Data":"e6c168f6f0c05e4f5eb90a292d5a2c1b4da6c9aba26db1e35dbbedf436c331de"} Nov 01 22:32:14.529962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:14.529507 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" event={"ID":"2fbf50c1-3598-470d-b539-7ac3dfdf0956","Type":"ContainerStarted","Data":"22ee6968132baedcef5ad7c9a90bbf7e3e7542350ec874f2686da876214a6fb0"} Nov 01 22:32:14.538947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:14.538918 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:32:14.549891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:14.549839 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podStartSLOduration=1.549822388 podStartE2EDuration="1.549822388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:14.548598415 +0000 UTC m=+547.197097653" watchObservedRunningTime="2026-11-01 22:32:14.549822388 +0000 UTC m=+547.198321625" Nov 01 22:32:14.654259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:14.654233 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:15.588455 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:32:15.588410 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:32:23.048572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.048488 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx"] Nov 01 22:32:23.053223 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.053200 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.060219 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.060192 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx"] Nov 01 22:32:23.068225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.068193 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-aa441b\"/\"premium-w5-aa441b-kserve-self-signed-certs\"" Nov 01 22:32:23.183494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.183463 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/239a637b-db11-45f9-a13b-2739c43b99e5-tls-certs\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.183686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.183507 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4nl\" (UniqueName: \"kubernetes.io/projected/239a637b-db11-45f9-a13b-2739c43b99e5-kube-api-access-tn4nl\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.183686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.183647 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-home\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.183838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.183821 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-model-cache\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.186307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.186271 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-dshm\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.186307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.186326 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-tmp-dir\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.287225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.287184 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/239a637b-db11-45f9-a13b-2739c43b99e5-tls-certs\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.287416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.287247 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4nl\" (UniqueName: \"kubernetes.io/projected/239a637b-db11-45f9-a13b-2739c43b99e5-kube-api-access-tn4nl\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.287416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.287294 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-home\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.287416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.287347 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-model-cache\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.287416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.287405 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-dshm\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.287635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.287440 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-tmp-dir\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.287809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.287780 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-home\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.287965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.287816 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-model-cache\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.288055 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.287877 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-tmp-dir\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.289677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.289648 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-dshm\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.289894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.289875 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/239a637b-db11-45f9-a13b-2739c43b99e5-tls-certs\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.295768 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.295749 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4nl\" (UniqueName: \"kubernetes.io/projected/239a637b-db11-45f9-a13b-2739c43b99e5-kube-api-access-tn4nl\") pod \"premium-w5-aa441b-kserve-66c7cbd7bb-m88cx\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.367475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.367408 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:23.641579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.641549 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx"] Nov 01 22:32:23.643510 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:32:23.643473 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod239a637b_db11_45f9_a13b_2739c43b99e5.slice/crio-355d8b05d6ef5466de3340c9912a60b5278ee6ebce4086d20e9ad6ee80da741d WatchSource:0}: Error finding container 355d8b05d6ef5466de3340c9912a60b5278ee6ebce4086d20e9ad6ee80da741d: Status 404 returned error can't find the container with id 355d8b05d6ef5466de3340c9912a60b5278ee6ebce4086d20e9ad6ee80da741d Nov 01 22:32:23.653809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.653785 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:23.668394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:23.668370 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:24.607658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:24.607623 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" event={"ID":"239a637b-db11-45f9-a13b-2739c43b99e5","Type":"ContainerStarted","Data":"6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596"} Nov 01 22:32:24.608073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:24.607666 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" event={"ID":"239a637b-db11-45f9-a13b-2739c43b99e5","Type":"ContainerStarted","Data":"355d8b05d6ef5466de3340c9912a60b5278ee6ebce4086d20e9ad6ee80da741d"} Nov 01 22:32:24.621607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:24.621579 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:32:24.626411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:24.626369 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podStartSLOduration=1.626358414 podStartE2EDuration="1.626358414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:24.623586593 +0000 UTC m=+557.272085829" watchObservedRunningTime="2026-11-01 22:32:24.626358414 +0000 UTC m=+557.274857650" Nov 01 22:32:25.368259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:25.368226 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:26.108582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.108545 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn"] Nov 01 22:32:26.112213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.112187 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.121227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.121072 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-381659\"/\"premium-w1-381659-kserve-self-signed-certs\"" Nov 01 22:32:26.124474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.124450 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn"] Nov 01 22:32:26.221546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.221519 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-model-cache\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.221546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.221545 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tmp-dir\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.221723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.221567 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-home\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.221723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.221683 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tls-certs\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.221789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.221752 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlscn\" (UniqueName: \"kubernetes.io/projected/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-kube-api-access-qlscn\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.221842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.221830 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-dshm\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.322553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.322518 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tls-certs\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.322553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.322557 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlscn\" (UniqueName: \"kubernetes.io/projected/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-kube-api-access-qlscn\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.322773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.322617 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-dshm\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.322773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.322674 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-model-cache\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.322773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.322699 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tmp-dir\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.322773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.322726 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-home\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.323104 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.323083 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-model-cache\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.323221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.323161 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-home\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.323221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.323191 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tmp-dir\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.324940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.324919 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-dshm\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.325256 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.325236 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tls-certs\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.330368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.330347 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlscn\" (UniqueName: \"kubernetes.io/projected/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-kube-api-access-qlscn\") pod \"premium-w1-381659-kserve-657d99b85f-9xjpn\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.428204 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.428125 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:26.705696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:26.705669 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn"] Nov 01 22:32:26.706660 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:32:26.706633 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1263f31e_87e9_4a4a_9f92_f0a03d3e0095.slice/crio-eaff3f3ed3e81e24a5f655292ca716742c0848417ca139f6492a9fd2bacd17ee WatchSource:0}: Error finding container eaff3f3ed3e81e24a5f655292ca716742c0848417ca139f6492a9fd2bacd17ee: Status 404 returned error can't find the container with id eaff3f3ed3e81e24a5f655292ca716742c0848417ca139f6492a9fd2bacd17ee Nov 01 22:32:27.629306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:27.629274 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" event={"ID":"1263f31e-87e9-4a4a-9f92-f0a03d3e0095","Type":"ContainerStarted","Data":"1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e"} Nov 01 22:32:27.629680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:27.629318 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" event={"ID":"1263f31e-87e9-4a4a-9f92-f0a03d3e0095","Type":"ContainerStarted","Data":"eaff3f3ed3e81e24a5f655292ca716742c0848417ca139f6492a9fd2bacd17ee"} Nov 01 22:32:27.646743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:27.646683 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podStartSLOduration=1.646668777 podStartE2EDuration="1.646668777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:27.645101638 +0000 UTC m=+560.293600874" watchObservedRunningTime="2026-11-01 22:32:27.646668777 +0000 UTC m=+560.295168011" Nov 01 22:32:28.433137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:28.428225 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:33.368149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:33.368094 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:33.380915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:33.380884 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:33.697261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:33.697182 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:32:36.428877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:36.428845 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:36.441699 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:36.441675 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:36.714633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:36.714553 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:32:44.561012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.560975 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8"] Nov 01 22:32:44.565504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.565486 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.570537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.570511 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:32:44.570708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.570536 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:32:44.570800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.570577 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:32:44.570800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.570578 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:32:44.574716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.574214 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8"] Nov 01 22:32:44.620386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.620356 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e55f0f1-jvqf8\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.620554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.620419 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzm8t\" (UniqueName: \"kubernetes.io/projected/7dc07b68-c585-4093-a1a6-ce9bec6b354a-kube-api-access-lzm8t\") pod \"maas-api-revoke-sub-4e55f0f1-jvqf8\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.620554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.620540 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e55f0f1-jvqf8\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.721614 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.721576 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e55f0f1-jvqf8\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.721776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.721665 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e55f0f1-jvqf8\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.721776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.721714 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzm8t\" (UniqueName: \"kubernetes.io/projected/7dc07b68-c585-4093-a1a6-ce9bec6b354a-kube-api-access-lzm8t\") pod \"maas-api-revoke-sub-4e55f0f1-jvqf8\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.722238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.722216 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e55f0f1-jvqf8\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.724038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.724012 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e55f0f1-jvqf8\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.729189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.729165 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzm8t\" (UniqueName: \"kubernetes.io/projected/7dc07b68-c585-4093-a1a6-ce9bec6b354a-kube-api-access-lzm8t\") pod \"maas-api-revoke-sub-4e55f0f1-jvqf8\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:44.881099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:44.881027 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:45.153608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:45.153577 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8"] Nov 01 22:32:45.159660 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:32:45.159634 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc07b68_c585_4093_a1a6_ce9bec6b354a.slice/crio-b3cbe43297a7d4a4011a72a61bddf9763e9b6a4b9573a7328c5a70d7975fdeb2 WatchSource:0}: Error finding container b3cbe43297a7d4a4011a72a61bddf9763e9b6a4b9573a7328c5a70d7975fdeb2: Status 404 returned error can't find the container with id b3cbe43297a7d4a4011a72a61bddf9763e9b6a4b9573a7328c5a70d7975fdeb2 Nov 01 22:32:45.590013 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:32:45.589976 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:32:45.773945 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:45.773906 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" event={"ID":"7dc07b68-c585-4093-a1a6-ce9bec6b354a","Type":"ContainerStarted","Data":"29d9c58ec62c015448351608e1b71866bf24d87b06f2e222f4ec2675547dd6db"} Nov 01 22:32:45.774126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:45.773964 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" event={"ID":"7dc07b68-c585-4093-a1a6-ce9bec6b354a","Type":"ContainerStarted","Data":"b3cbe43297a7d4a4011a72a61bddf9763e9b6a4b9573a7328c5a70d7975fdeb2"} Nov 01 22:32:45.790318 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:45.790270 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" podStartSLOduration=1.790255137 podStartE2EDuration="1.790255137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:45.788530183 +0000 UTC m=+578.437029417" watchObservedRunningTime="2026-11-01 22:32:45.790255137 +0000 UTC m=+578.438754373" Nov 01 22:32:46.785147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:46.784150 6287 generic.go:356] "Generic (PLEG): container finished" podID="7dc07b68-c585-4093-a1a6-ce9bec6b354a" containerID="29d9c58ec62c015448351608e1b71866bf24d87b06f2e222f4ec2675547dd6db" exitCode=0 Nov 01 22:32:46.785147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:46.784331 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" event={"ID":"7dc07b68-c585-4093-a1a6-ce9bec6b354a","Type":"ContainerDied","Data":"29d9c58ec62c015448351608e1b71866bf24d87b06f2e222f4ec2675547dd6db"} Nov 01 22:32:47.936270 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:47.936244 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:47.960108 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:47.960072 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc07b68-c585-4093-a1a6-ce9bec6b354a-kube-api-access-lzm8t\" (UniqueName: \"kubernetes.io/projected/7dc07b68-c585-4093-a1a6-ce9bec6b354a-kube-api-access-lzm8t\") pod \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " Nov 01 22:32:47.960296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:47.960155 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-xks-ca\") pod \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " Nov 01 22:32:47.960296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:47.960225 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-openshift-ca\") pod \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\" (UID: \"7dc07b68-c585-4093-a1a6-ce9bec6b354a\") " Nov 01 22:32:47.960624 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:47.960599 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-openshift-ca" pod "7dc07b68-c585-4093-a1a6-ce9bec6b354a" (UID: "7dc07b68-c585-4093-a1a6-ce9bec6b354a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:47.960765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:47.960750 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:47.962312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:47.962285 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc07b68-c585-4093-a1a6-ce9bec6b354a-kube-api-access-lzm8t" pod "7dc07b68-c585-4093-a1a6-ce9bec6b354a" (UID: "7dc07b68-c585-4093-a1a6-ce9bec6b354a"). InnerVolumeSpecName "kube-api-access-lzm8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:47.962428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:47.962404 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-xks-ca" pod "7dc07b68-c585-4093-a1a6-ce9bec6b354a" (UID: "7dc07b68-c585-4093-a1a6-ce9bec6b354a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:48.062151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:48.062042 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzm8t\" (UniqueName: \"kubernetes.io/projected/7dc07b68-c585-4093-a1a6-ce9bec6b354a-kube-api-access-lzm8t\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:48.062151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:48.062070 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc07b68-c585-4093-a1a6-ce9bec6b354a-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:48.796880 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:48.796849 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" Nov 01 22:32:48.797050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:48.796853 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8" event={"ID":"7dc07b68-c585-4093-a1a6-ce9bec6b354a","Type":"ContainerDied","Data":"b3cbe43297a7d4a4011a72a61bddf9763e9b6a4b9573a7328c5a70d7975fdeb2"} Nov 01 22:32:48.797050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:48.796997 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3cbe43297a7d4a4011a72a61bddf9763e9b6a4b9573a7328c5a70d7975fdeb2" Nov 01 22:32:58.401725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.401689 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4"] Nov 01 22:32:58.402238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.402215 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7dc07b68-c585-4093-a1a6-ce9bec6b354a" containerName="revoke-keys" Nov 01 22:32:58.402238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.402239 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc07b68-c585-4093-a1a6-ce9bec6b354a" containerName="revoke-keys" Nov 01 22:32:58.402327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.402314 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc07b68-c585-4093-a1a6-ce9bec6b354a" containerName="revoke-keys" Nov 01 22:32:58.405654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.405632 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.412665 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.412640 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:32:58.412804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.412724 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:32:58.412804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.412790 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:32:58.412917 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.412835 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:32:58.415290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.414730 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4"] Nov 01 22:32:58.477846 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.477816 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9889\" (UniqueName: \"kubernetes.io/projected/dea21ba9-723d-4e2e-b4e6-88eab684fa57-kube-api-access-j9889\") pod \"maas-api-revoke-sub-07daabac-g4ds4\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.478018 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.477940 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07daabac-g4ds4\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.478018 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.478003 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07daabac-g4ds4\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.535512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.535482 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch"] Nov 01 22:32:58.539693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.539675 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.546775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.546746 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch"] Nov 01 22:32:58.579462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.579415 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9889\" (UniqueName: \"kubernetes.io/projected/dea21ba9-723d-4e2e-b4e6-88eab684fa57-kube-api-access-j9889\") pod \"maas-api-revoke-sub-07daabac-g4ds4\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.579462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.579520 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07daabac-g4ds4\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.579462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.579576 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07daabac-g4ds4\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.580456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.580431 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07daabac-g4ds4\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.582178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.582143 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07daabac-g4ds4\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.594012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.593985 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9889\" (UniqueName: \"kubernetes.io/projected/dea21ba9-723d-4e2e-b4e6-88eab684fa57-kube-api-access-j9889\") pod \"maas-api-revoke-sub-07daabac-g4ds4\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.680683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.680602 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ac5f79-qqjch\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.680683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.680639 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzb2x\" (UniqueName: \"kubernetes.io/projected/e9874117-25ff-4db8-b1ca-2cdd903c1e11-kube-api-access-jzb2x\") pod \"maas-api-revoke-sub-24ac5f79-qqjch\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.680878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.680840 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ac5f79-qqjch\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.723017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.722986 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:32:58.782059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.782028 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ac5f79-qqjch\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.782253 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.782096 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ac5f79-qqjch\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.782253 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.782143 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzb2x\" (UniqueName: \"kubernetes.io/projected/e9874117-25ff-4db8-b1ca-2cdd903c1e11-kube-api-access-jzb2x\") pod \"maas-api-revoke-sub-24ac5f79-qqjch\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.784484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.784458 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ac5f79-qqjch\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.787525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.787494 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ac5f79-qqjch\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.797546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.797520 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzb2x\" (UniqueName: \"kubernetes.io/projected/e9874117-25ff-4db8-b1ca-2cdd903c1e11-kube-api-access-jzb2x\") pod \"maas-api-revoke-sub-24ac5f79-qqjch\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.861505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.861467 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:32:58.924033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.923981 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz"] Nov 01 22:32:58.941998 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.934247 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:58.946355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.943861 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-381659-openshift-default-dockercfg-wjt66\"" Nov 01 22:32:58.951889 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:58.946875 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz"] Nov 01 22:32:59.001215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.001187 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4"] Nov 01 22:32:59.003517 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:32:59.003482 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea21ba9_723d_4e2e_b4e6_88eab684fa57.slice/crio-f7f022e70920218972b23780c9b21d7d831a0d67900ea8539ecd7dddeaa2981b WatchSource:0}: Error finding container f7f022e70920218972b23780c9b21d7d831a0d67900ea8539ecd7dddeaa2981b: Status 404 returned error can't find the container with id f7f022e70920218972b23780c9b21d7d831a0d67900ea8539ecd7dddeaa2981b Nov 01 22:32:59.085926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.085886 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.086107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.085959 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.086107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.086081 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.086270 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.086169 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.086270 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.086238 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.086386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.086311 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhh5\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-kube-api-access-cfhh5\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.086386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.086348 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.086509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.086488 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.086571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.086528 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.096998 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.096967 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg"] Nov 01 22:32:59.102846 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.102823 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.109150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.109100 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3d8490-openshift-default-dockercfg-vp9kv\"" Nov 01 22:32:59.114205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.114183 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg"] Nov 01 22:32:59.151746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.151712 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch"] Nov 01 22:32:59.152716 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:32:59.152684 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9874117_25ff_4db8_b1ca_2cdd903c1e11.slice/crio-ff8359b6f228ad4afbde38f6fc4642d402e67f4d4975ec8a1c1813f0e08f80ec WatchSource:0}: Error finding container ff8359b6f228ad4afbde38f6fc4642d402e67f4d4975ec8a1c1813f0e08f80ec: Status 404 returned error can't find the container with id ff8359b6f228ad4afbde38f6fc4642d402e67f4d4975ec8a1c1813f0e08f80ec Nov 01 22:32:59.187725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.187696 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.187866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.187745 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.187866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.187791 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.187866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.187825 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.188053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.187875 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.188053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.187909 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhh5\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-kube-api-access-cfhh5\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.188053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.187945 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.188053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.187982 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.188053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188015 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03620ee4-ac0f-435c-a4d3-011c85212538-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.188359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188057 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03620ee4-ac0f-435c-a4d3-011c85212538-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.188359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188080 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzrj\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-kube-api-access-hvzrj\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.188359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188133 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.188359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188163 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.188359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188208 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.188359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188258 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.188359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188297 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.188359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188356 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.188784 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188409 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.188936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.188831 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.189556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.189515 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.189556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.189534 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.190400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.190370 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.190666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.190640 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.192246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.191697 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.192819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.192796 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.202866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.200041 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.202866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.201946 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhh5\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-kube-api-access-cfhh5\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-t4vrz\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.267608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.267569 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:32:59.291778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.291745 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03620ee4-ac0f-435c-a4d3-011c85212538-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.291908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.291794 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzrj\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-kube-api-access-hvzrj\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.291908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.291851 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.292153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.292091 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.292295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.292273 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.292358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.292281 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.292442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.292420 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.292513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.292489 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.292584 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.292568 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.292647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.292605 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03620ee4-ac0f-435c-a4d3-011c85212538-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.292647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.292623 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.292832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.292800 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.293389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.293359 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03620ee4-ac0f-435c-a4d3-011c85212538-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.293573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.293542 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.295866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.295764 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.300866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.300847 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03620ee4-ac0f-435c-a4d3-011c85212538-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.303950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.303924 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzrj\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-kube-api-access-hvzrj\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.309976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.309954 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.420024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.419990 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:32:59.574272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.572360 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz"] Nov 01 22:32:59.574272 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:32:59.573702 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45cef72_e50b_4fe7_9a7c_d4b52a5f0c9f.slice/crio-350d872f694e9bc884963de1785d48c513cbd7fddbe50bc9a5894e8f8c27596f WatchSource:0}: Error finding container 350d872f694e9bc884963de1785d48c513cbd7fddbe50bc9a5894e8f8c27596f: Status 404 returned error can't find the container with id 350d872f694e9bc884963de1785d48c513cbd7fddbe50bc9a5894e8f8c27596f Nov 01 22:32:59.576498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.576459 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:32:59.576620 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.576538 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:32:59.576620 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.576584 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:32:59.730485 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.730455 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg"] Nov 01 22:32:59.733068 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:32:59.733041 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03620ee4_ac0f_435c_a4d3_011c85212538.slice/crio-d7301ed3bafe18dca00305f2bcbf77fd692e8d75fe7d1468d6ad08b0ce10e0a6 WatchSource:0}: Error finding container d7301ed3bafe18dca00305f2bcbf77fd692e8d75fe7d1468d6ad08b0ce10e0a6: Status 404 returned error can't find the container with id d7301ed3bafe18dca00305f2bcbf77fd692e8d75fe7d1468d6ad08b0ce10e0a6 Nov 01 22:32:59.736143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.736083 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:32:59.736277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.736175 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:32:59.736277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.736227 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:32:59.888417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.887306 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" event={"ID":"03620ee4-ac0f-435c-a4d3-011c85212538","Type":"ContainerStarted","Data":"f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd"} Nov 01 22:32:59.888417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.887361 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" event={"ID":"03620ee4-ac0f-435c-a4d3-011c85212538","Type":"ContainerStarted","Data":"d7301ed3bafe18dca00305f2bcbf77fd692e8d75fe7d1468d6ad08b0ce10e0a6"} Nov 01 22:32:59.890468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.890437 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" event={"ID":"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f","Type":"ContainerStarted","Data":"4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3"} Nov 01 22:32:59.890587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.890503 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" event={"ID":"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f","Type":"ContainerStarted","Data":"350d872f694e9bc884963de1785d48c513cbd7fddbe50bc9a5894e8f8c27596f"} Nov 01 22:32:59.894305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.893699 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" event={"ID":"e9874117-25ff-4db8-b1ca-2cdd903c1e11","Type":"ContainerStarted","Data":"19003558e4e20785aa920409ad820573978d3fcc57b3297cec8451aa2a2f6fb9"} Nov 01 22:32:59.894305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.893745 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" event={"ID":"e9874117-25ff-4db8-b1ca-2cdd903c1e11","Type":"ContainerStarted","Data":"ff8359b6f228ad4afbde38f6fc4642d402e67f4d4975ec8a1c1813f0e08f80ec"} Nov 01 22:32:59.896148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.896086 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" event={"ID":"dea21ba9-723d-4e2e-b4e6-88eab684fa57","Type":"ContainerStarted","Data":"bc405dd19931f1a4c3cb847dbba0195750e4f3b33e00efec89d45911e4ff1e33"} Nov 01 22:32:59.896544 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.896521 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" event={"ID":"dea21ba9-723d-4e2e-b4e6-88eab684fa57","Type":"ContainerStarted","Data":"f7f022e70920218972b23780c9b21d7d831a0d67900ea8539ecd7dddeaa2981b"} Nov 01 22:32:59.910009 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.909585 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podStartSLOduration=0.909570884 podStartE2EDuration="909.570884ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:59.907783392 +0000 UTC m=+592.556282644" watchObservedRunningTime="2026-11-01 22:32:59.909570884 +0000 UTC m=+592.558070120" Nov 01 22:32:59.924776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.924700 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" podStartSLOduration=1.9246792560000001 podStartE2EDuration="1.924679256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:59.923248373 +0000 UTC m=+592.571747609" watchObservedRunningTime="2026-11-01 22:32:59.924679256 +0000 UTC m=+592.573178492" Nov 01 22:32:59.941773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.941713 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" podStartSLOduration=1.941673512 podStartE2EDuration="1.941673512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:59.940596312 +0000 UTC m=+592.589095547" watchObservedRunningTime="2026-11-01 22:32:59.941673512 +0000 UTC m=+592.590172748" Nov 01 22:32:59.965163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:32:59.964278 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podStartSLOduration=1.9642581080000001 podStartE2EDuration="1.964258108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:59.963219682 +0000 UTC m=+592.611718918" watchObservedRunningTime="2026-11-01 22:32:59.964258108 +0000 UTC m=+592.612757344" Nov 01 22:33:00.274136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.271912 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:33:00.282265 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.282229 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:33:00.282532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.282506 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 22:33:00.421569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.421487 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:33:00.426639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.426245 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:33:00.426639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.426307 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:33:00.929761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.929261 6287 generic.go:356] "Generic (PLEG): container finished" podID="e9874117-25ff-4db8-b1ca-2cdd903c1e11" containerID="19003558e4e20785aa920409ad820573978d3fcc57b3297cec8451aa2a2f6fb9" exitCode=0 Nov 01 22:33:00.929761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.929365 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" event={"ID":"e9874117-25ff-4db8-b1ca-2cdd903c1e11","Type":"ContainerDied","Data":"19003558e4e20785aa920409ad820573978d3fcc57b3297cec8451aa2a2f6fb9"} Nov 01 22:33:00.947389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.945702 6287 generic.go:356] "Generic (PLEG): container finished" podID="dea21ba9-723d-4e2e-b4e6-88eab684fa57" containerID="bc405dd19931f1a4c3cb847dbba0195750e4f3b33e00efec89d45911e4ff1e33" exitCode=0 Nov 01 22:33:00.947389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:00.946689 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" event={"ID":"dea21ba9-723d-4e2e-b4e6-88eab684fa57","Type":"ContainerDied","Data":"bc405dd19931f1a4c3cb847dbba0195750e4f3b33e00efec89d45911e4ff1e33"} Nov 01 22:33:01.268848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:01.268809 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:33:01.269058 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:01.268862 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 22:33:01.423550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:01.423092 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:33:01.423550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:01.423173 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:33:02.263352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.260338 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:33:02.269530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.269172 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:33:02.269530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.269226 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 22:33:02.337175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.337140 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea21ba9-723d-4e2e-b4e6-88eab684fa57-kube-api-access-j9889\" (UniqueName: \"kubernetes.io/projected/dea21ba9-723d-4e2e-b4e6-88eab684fa57-kube-api-access-j9889\") pod \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " Nov 01 22:33:02.337359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.337305 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-openshift-ca\") pod \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " Nov 01 22:33:02.337424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.337408 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-xks-ca\") pod \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\" (UID: \"dea21ba9-723d-4e2e-b4e6-88eab684fa57\") " Nov 01 22:33:02.338900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.338185 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:33:02.344152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.340149 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-xks-ca" pod "dea21ba9-723d-4e2e-b4e6-88eab684fa57" (UID: "dea21ba9-723d-4e2e-b4e6-88eab684fa57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:02.344152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.340394 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-openshift-ca" pod "dea21ba9-723d-4e2e-b4e6-88eab684fa57" (UID: "dea21ba9-723d-4e2e-b4e6-88eab684fa57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:02.346422 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.346289 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea21ba9-723d-4e2e-b4e6-88eab684fa57-kube-api-access-j9889" pod "dea21ba9-723d-4e2e-b4e6-88eab684fa57" (UID: "dea21ba9-723d-4e2e-b4e6-88eab684fa57"). InnerVolumeSpecName "kube-api-access-j9889". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:02.427150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.426516 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:33:02.427150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.426594 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:33:02.438931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.438907 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9874117-25ff-4db8-b1ca-2cdd903c1e11-kube-api-access-jzb2x\" (UniqueName: \"kubernetes.io/projected/e9874117-25ff-4db8-b1ca-2cdd903c1e11-kube-api-access-jzb2x\") pod \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " Nov 01 22:33:02.439211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.439197 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-openshift-ca\") pod \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " Nov 01 22:33:02.439348 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.439335 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-xks-ca\") pod \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\" (UID: \"e9874117-25ff-4db8-b1ca-2cdd903c1e11\") " Nov 01 22:33:02.439735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.439713 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:02.440133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.439826 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-openshift-ca" pod "e9874117-25ff-4db8-b1ca-2cdd903c1e11" (UID: "e9874117-25ff-4db8-b1ca-2cdd903c1e11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:02.440133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.439878 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea21ba9-723d-4e2e-b4e6-88eab684fa57-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:02.440133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.439902 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9889\" (UniqueName: \"kubernetes.io/projected/dea21ba9-723d-4e2e-b4e6-88eab684fa57-kube-api-access-j9889\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:02.441817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.441789 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9874117-25ff-4db8-b1ca-2cdd903c1e11-kube-api-access-jzb2x" pod "e9874117-25ff-4db8-b1ca-2cdd903c1e11" (UID: "e9874117-25ff-4db8-b1ca-2cdd903c1e11"). InnerVolumeSpecName "kube-api-access-jzb2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:02.442135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.442087 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-xks-ca" pod "e9874117-25ff-4db8-b1ca-2cdd903c1e11" (UID: "e9874117-25ff-4db8-b1ca-2cdd903c1e11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:02.540746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.540710 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:02.540746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.540741 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9874117-25ff-4db8-b1ca-2cdd903c1e11-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:02.540746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.540751 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzb2x\" (UniqueName: \"kubernetes.io/projected/e9874117-25ff-4db8-b1ca-2cdd903c1e11-kube-api-access-jzb2x\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:02.961603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.961509 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" event={"ID":"e9874117-25ff-4db8-b1ca-2cdd903c1e11","Type":"ContainerDied","Data":"ff8359b6f228ad4afbde38f6fc4642d402e67f4d4975ec8a1c1813f0e08f80ec"} Nov 01 22:33:02.961603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.961567 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8359b6f228ad4afbde38f6fc4642d402e67f4d4975ec8a1c1813f0e08f80ec" Nov 01 22:33:02.961827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.961642 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch" Nov 01 22:33:02.963868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.963833 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" event={"ID":"dea21ba9-723d-4e2e-b4e6-88eab684fa57","Type":"ContainerDied","Data":"f7f022e70920218972b23780c9b21d7d831a0d67900ea8539ecd7dddeaa2981b"} Nov 01 22:33:02.964003 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.963889 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7f022e70920218972b23780c9b21d7d831a0d67900ea8539ecd7dddeaa2981b" Nov 01 22:33:02.964003 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:02.963948 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4" Nov 01 22:33:03.268808 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:03.268763 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:33:03.268963 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:03.268831 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 22:33:03.421137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:03.420478 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:33:03.421137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:03.420541 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:33:04.268393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:04.268342 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:33:04.268874 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:04.268432 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 22:33:04.420916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:04.420468 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:33:04.420916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:04.420548 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:33:05.268394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:05.268359 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 22:33:05.268556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:05.268409 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 22:33:05.421227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:05.421182 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:33:05.421405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:05.421248 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:33:06.460655 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:06.460621 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:33:06.461942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:06.461920 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:33:06.463007 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:06.462984 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:33:06.635344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:06.635302 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:33:06.636279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:06.636209 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:33:06.637380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:06.637359 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:33:10.389973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.389933 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf"] Nov 01 22:33:10.390635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.390614 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e9874117-25ff-4db8-b1ca-2cdd903c1e11" containerName="revoke-keys" Nov 01 22:33:10.390721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.390637 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9874117-25ff-4db8-b1ca-2cdd903c1e11" containerName="revoke-keys" Nov 01 22:33:10.390721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.390669 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dea21ba9-723d-4e2e-b4e6-88eab684fa57" containerName="revoke-keys" Nov 01 22:33:10.390721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.390678 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea21ba9-723d-4e2e-b4e6-88eab684fa57" containerName="revoke-keys" Nov 01 22:33:10.390885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.390829 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9874117-25ff-4db8-b1ca-2cdd903c1e11" containerName="revoke-keys" Nov 01 22:33:10.390885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.390846 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea21ba9-723d-4e2e-b4e6-88eab684fa57" containerName="revoke-keys" Nov 01 22:33:10.395364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.395342 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.400542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.400512 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:33:10.400660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.400626 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf"] Nov 01 22:33:10.401023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.400742 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:33:10.401023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.400746 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:33:10.401023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.400781 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:33:10.550544 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.550511 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036b3107-6lkjf\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.550707 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.550632 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036b3107-6lkjf\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.550775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.550715 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5xp8\" (UniqueName: \"kubernetes.io/projected/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-kube-api-access-b5xp8\") pod \"maas-api-revoke-sub-036b3107-6lkjf\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.652344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.652267 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036b3107-6lkjf\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.652498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.652384 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036b3107-6lkjf\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.652498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.652446 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5xp8\" (UniqueName: \"kubernetes.io/projected/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-kube-api-access-b5xp8\") pod \"maas-api-revoke-sub-036b3107-6lkjf\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.653045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.652991 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-036b3107-6lkjf\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.655143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.655091 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-036b3107-6lkjf\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.660768 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.660746 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5xp8\" (UniqueName: \"kubernetes.io/projected/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-kube-api-access-b5xp8\") pod \"maas-api-revoke-sub-036b3107-6lkjf\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.710429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.710403 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:10.993426 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:10.993394 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aca02aa_8a09_4dc5_b2ad_2fc6eacaaf7c.slice/crio-6f8c5376018507924e2a5725f0c8a9e4395198f0904898caefb6f013cbaf3d87 WatchSource:0}: Error finding container 6f8c5376018507924e2a5725f0c8a9e4395198f0904898caefb6f013cbaf3d87: Status 404 returned error can't find the container with id 6f8c5376018507924e2a5725f0c8a9e4395198f0904898caefb6f013cbaf3d87 Nov 01 22:33:10.993736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:10.993708 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf"] Nov 01 22:33:11.058183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.058154 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" event={"ID":"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c","Type":"ContainerStarted","Data":"6f8c5376018507924e2a5725f0c8a9e4395198f0904898caefb6f013cbaf3d87"} Nov 01 22:33:11.693953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.693877 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s"] Nov 01 22:33:11.697915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.697896 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:11.705156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.705132 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s"] Nov 01 22:33:11.767983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.767951 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhvkj\" (UniqueName: \"kubernetes.io/projected/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-kube-api-access-nhvkj\") pod \"maas-api-revoke-sub-82f67b0c-5pm4s\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:11.768176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.768011 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82f67b0c-5pm4s\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:11.768262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.768179 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82f67b0c-5pm4s\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:11.869639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.869606 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82f67b0c-5pm4s\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:11.869822 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.869703 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82f67b0c-5pm4s\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:11.869872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.869826 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhvkj\" (UniqueName: \"kubernetes.io/projected/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-kube-api-access-nhvkj\") pod \"maas-api-revoke-sub-82f67b0c-5pm4s\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:11.870590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.870565 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82f67b0c-5pm4s\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:11.872612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.872583 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82f67b0c-5pm4s\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:11.883651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:11.883627 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhvkj\" (UniqueName: \"kubernetes.io/projected/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-kube-api-access-nhvkj\") pod \"maas-api-revoke-sub-82f67b0c-5pm4s\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:12.022146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:12.021316 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:12.068233 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:12.068197 6287 generic.go:356] "Generic (PLEG): container finished" podID="3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c" containerID="44646e8073b3c49d11489952e9d2f8f3e921e24c2c59e2419e5b1de03a4b6004" exitCode=0 Nov 01 22:33:12.068372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:12.068233 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" event={"ID":"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c","Type":"ContainerDied","Data":"44646e8073b3c49d11489952e9d2f8f3e921e24c2c59e2419e5b1de03a4b6004"} Nov 01 22:33:12.317424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:12.317390 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s"] Nov 01 22:33:12.322094 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:12.322059 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a48b7e_1c02_4f3a_8e14_6e4ac755011b.slice/crio-085fbe3a889074f77514d54d11c65fdcb661ea4c1906d020dbdf8f9226367cae WatchSource:0}: Error finding container 085fbe3a889074f77514d54d11c65fdcb661ea4c1906d020dbdf8f9226367cae: Status 404 returned error can't find the container with id 085fbe3a889074f77514d54d11c65fdcb661ea4c1906d020dbdf8f9226367cae Nov 01 22:33:13.076975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.076936 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" event={"ID":"82a48b7e-1c02-4f3a-8e14-6e4ac755011b","Type":"ContainerStarted","Data":"8ab2e8b348ac5c76682b39f5f1d956f371d5f6da46d974a57505c826547d246a"} Nov 01 22:33:13.077408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.076997 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" event={"ID":"82a48b7e-1c02-4f3a-8e14-6e4ac755011b","Type":"ContainerStarted","Data":"085fbe3a889074f77514d54d11c65fdcb661ea4c1906d020dbdf8f9226367cae"} Nov 01 22:33:13.096012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.095963 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" podStartSLOduration=2.095948937 podStartE2EDuration="2.095948937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:13.09354818 +0000 UTC m=+605.742047414" watchObservedRunningTime="2026-11-01 22:33:13.095948937 +0000 UTC m=+605.744448172" Nov 01 22:33:13.234787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.234763 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:13.411399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.411377 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-kube-api-access-b5xp8\" (UniqueName: \"kubernetes.io/projected/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-kube-api-access-b5xp8\") pod \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " Nov 01 22:33:13.411507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.411474 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-xks-ca\") pod \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " Nov 01 22:33:13.411588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.411575 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-openshift-ca\") pod \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\" (UID: \"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c\") " Nov 01 22:33:13.411943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.411912 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-openshift-ca" pod "3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c" (UID: "3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:13.413343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.413319 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-kube-api-access-b5xp8" pod "3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c" (UID: "3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c"). InnerVolumeSpecName "kube-api-access-b5xp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:13.413556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.413534 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-xks-ca" pod "3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c" (UID: "3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:13.512965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.512937 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5xp8\" (UniqueName: \"kubernetes.io/projected/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-kube-api-access-b5xp8\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:13.512965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.512963 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:13.513154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:13.512974 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:14.085020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:14.084979 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" Nov 01 22:33:14.085531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:14.085023 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf" event={"ID":"3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c","Type":"ContainerDied","Data":"6f8c5376018507924e2a5725f0c8a9e4395198f0904898caefb6f013cbaf3d87"} Nov 01 22:33:14.085531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:14.085069 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f8c5376018507924e2a5725f0c8a9e4395198f0904898caefb6f013cbaf3d87" Nov 01 22:33:14.087225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:14.087190 6287 generic.go:356] "Generic (PLEG): container finished" podID="82a48b7e-1c02-4f3a-8e14-6e4ac755011b" containerID="8ab2e8b348ac5c76682b39f5f1d956f371d5f6da46d974a57505c826547d246a" exitCode=0 Nov 01 22:33:14.087366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:14.087265 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" event={"ID":"82a48b7e-1c02-4f3a-8e14-6e4ac755011b","Type":"ContainerDied","Data":"8ab2e8b348ac5c76682b39f5f1d956f371d5f6da46d974a57505c826547d246a"} Nov 01 22:33:15.325902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.325594 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:15.448184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.447203 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-kube-api-access-nhvkj\" (UniqueName: \"kubernetes.io/projected/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-kube-api-access-nhvkj\") pod \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " Nov 01 22:33:15.448184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.447378 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-xks-ca\") pod \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " Nov 01 22:33:15.448184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.447415 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-openshift-ca\") pod \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\" (UID: \"82a48b7e-1c02-4f3a-8e14-6e4ac755011b\") " Nov 01 22:33:15.448184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.448151 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-openshift-ca" pod "82a48b7e-1c02-4f3a-8e14-6e4ac755011b" (UID: "82a48b7e-1c02-4f3a-8e14-6e4ac755011b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:15.497967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.493633 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-xks-ca" pod "82a48b7e-1c02-4f3a-8e14-6e4ac755011b" (UID: "82a48b7e-1c02-4f3a-8e14-6e4ac755011b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:15.503389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.503317 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-kube-api-access-nhvkj" pod "82a48b7e-1c02-4f3a-8e14-6e4ac755011b" (UID: "82a48b7e-1c02-4f3a-8e14-6e4ac755011b"). InnerVolumeSpecName "kube-api-access-nhvkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:15.561142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.549394 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:15.561142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.549430 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:15.561142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.549461 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhvkj\" (UniqueName: \"kubernetes.io/projected/82a48b7e-1c02-4f3a-8e14-6e4ac755011b-kube-api-access-nhvkj\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:15.674188 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:33:15.674145 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:33:15.880719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.880690 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x"] Nov 01 22:33:15.881204 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.881190 6287 cpu_manager.go:414] "containerMap: removing container" podUID="3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c" containerName="revoke-keys" Nov 01 22:33:15.881255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.881205 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c" containerName="revoke-keys" Nov 01 22:33:15.881255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.881215 6287 cpu_manager.go:414] "containerMap: removing container" podUID="82a48b7e-1c02-4f3a-8e14-6e4ac755011b" containerName="revoke-keys" Nov 01 22:33:15.881255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.881220 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a48b7e-1c02-4f3a-8e14-6e4ac755011b" containerName="revoke-keys" Nov 01 22:33:15.881346 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.881304 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c" containerName="revoke-keys" Nov 01 22:33:15.881346 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.881313 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a48b7e-1c02-4f3a-8e14-6e4ac755011b" containerName="revoke-keys" Nov 01 22:33:15.884670 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.884655 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:15.893126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.893045 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x"] Nov 01 22:33:15.905691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.905660 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n"] Nov 01 22:33:15.910381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.910349 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:15.919518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.919496 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n"] Nov 01 22:33:15.954279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.954243 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhkk\" (UniqueName: \"kubernetes.io/projected/7de4ba9c-624a-4a12-9957-2778c9983213-kube-api-access-blhkk\") pod \"maas-api-revoke-sub-8ba2d8b2-j6v7x\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:15.954454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.954311 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9v8h\" (UniqueName: \"kubernetes.io/projected/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-kube-api-access-d9v8h\") pod \"maas-api-revoke-sub-99a1aea6-7g96n\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:15.954454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.954361 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba2d8b2-j6v7x\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:15.954454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.954418 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99a1aea6-7g96n\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:15.954618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.954510 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba2d8b2-j6v7x\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:15.954618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:15.954545 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99a1aea6-7g96n\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:16.055598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.055557 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhkk\" (UniqueName: \"kubernetes.io/projected/7de4ba9c-624a-4a12-9957-2778c9983213-kube-api-access-blhkk\") pod \"maas-api-revoke-sub-8ba2d8b2-j6v7x\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:16.055782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.055605 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9v8h\" (UniqueName: \"kubernetes.io/projected/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-kube-api-access-d9v8h\") pod \"maas-api-revoke-sub-99a1aea6-7g96n\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:16.055782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.055721 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba2d8b2-j6v7x\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:16.055932 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.055778 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99a1aea6-7g96n\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:16.055932 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.055852 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba2d8b2-j6v7x\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:16.055932 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.055872 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99a1aea6-7g96n\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:16.056525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.056500 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba2d8b2-j6v7x\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:16.056661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.056575 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99a1aea6-7g96n\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:16.058036 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.058007 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba2d8b2-j6v7x\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:16.058235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.058210 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99a1aea6-7g96n\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:16.062907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.062886 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhkk\" (UniqueName: \"kubernetes.io/projected/7de4ba9c-624a-4a12-9957-2778c9983213-kube-api-access-blhkk\") pod \"maas-api-revoke-sub-8ba2d8b2-j6v7x\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:16.063017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.063000 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9v8h\" (UniqueName: \"kubernetes.io/projected/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-kube-api-access-d9v8h\") pod \"maas-api-revoke-sub-99a1aea6-7g96n\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:16.100582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.100559 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" Nov 01 22:33:16.100684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.100594 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s" event={"ID":"82a48b7e-1c02-4f3a-8e14-6e4ac755011b","Type":"ContainerDied","Data":"085fbe3a889074f77514d54d11c65fdcb661ea4c1906d020dbdf8f9226367cae"} Nov 01 22:33:16.100684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.100662 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="085fbe3a889074f77514d54d11c65fdcb661ea4c1906d020dbdf8f9226367cae" Nov 01 22:33:16.201292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.201205 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:16.234826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.234798 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:16.512322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.512291 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x"] Nov 01 22:33:16.513045 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:16.513013 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de4ba9c_624a_4a12_9957_2778c9983213.slice/crio-3e406a1dde18df1aa8a9f843ccb4f89001b8510b169722ec603bae91f9ee4587 WatchSource:0}: Error finding container 3e406a1dde18df1aa8a9f843ccb4f89001b8510b169722ec603bae91f9ee4587: Status 404 returned error can't find the container with id 3e406a1dde18df1aa8a9f843ccb4f89001b8510b169722ec603bae91f9ee4587 Nov 01 22:33:16.550674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:16.550651 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n"] Nov 01 22:33:16.559287 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:16.559255 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378cb8d5_70b2_4dd3_aed1_61fc626cd5cc.slice/crio-c7d94b93b8e7a188d14c631aa7baa4143616b4450e860f7cf85ded67d4b7bf0f WatchSource:0}: Error finding container c7d94b93b8e7a188d14c631aa7baa4143616b4450e860f7cf85ded67d4b7bf0f: Status 404 returned error can't find the container with id c7d94b93b8e7a188d14c631aa7baa4143616b4450e860f7cf85ded67d4b7bf0f Nov 01 22:33:17.108551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:17.108464 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" event={"ID":"7de4ba9c-624a-4a12-9957-2778c9983213","Type":"ContainerStarted","Data":"acdbe1114d7c43f56e0402d45e878bd71a840d89cf031ae36d88bfabf3afd951"} Nov 01 22:33:17.108551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:17.108526 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" event={"ID":"7de4ba9c-624a-4a12-9957-2778c9983213","Type":"ContainerStarted","Data":"3e406a1dde18df1aa8a9f843ccb4f89001b8510b169722ec603bae91f9ee4587"} Nov 01 22:33:17.110159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:17.110101 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" event={"ID":"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc","Type":"ContainerStarted","Data":"a235e00caef896ea04d88cd008b15cbbf4173d5f4e73f9ab55b1d58099e7fe85"} Nov 01 22:33:17.110313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:17.110175 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" event={"ID":"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc","Type":"ContainerStarted","Data":"c7d94b93b8e7a188d14c631aa7baa4143616b4450e860f7cf85ded67d4b7bf0f"} Nov 01 22:33:17.125539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:17.125494 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" podStartSLOduration=2.12548135 podStartE2EDuration="2.12548135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:17.124333485 +0000 UTC m=+609.772832721" watchObservedRunningTime="2026-11-01 22:33:17.12548135 +0000 UTC m=+609.773980585" Nov 01 22:33:17.147021 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:17.146969 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" podStartSLOduration=2.146956094 podStartE2EDuration="2.146956094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:17.14481071 +0000 UTC m=+609.793309945" watchObservedRunningTime="2026-11-01 22:33:17.146956094 +0000 UTC m=+609.795455328" Nov 01 22:33:18.127091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.120374 6287 generic.go:356] "Generic (PLEG): container finished" podID="378cb8d5-70b2-4dd3-aed1-61fc626cd5cc" containerID="a235e00caef896ea04d88cd008b15cbbf4173d5f4e73f9ab55b1d58099e7fe85" exitCode=0 Nov 01 22:33:18.127091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.120542 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" event={"ID":"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc","Type":"ContainerDied","Data":"a235e00caef896ea04d88cd008b15cbbf4173d5f4e73f9ab55b1d58099e7fe85"} Nov 01 22:33:18.127091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.124916 6287 generic.go:356] "Generic (PLEG): container finished" podID="7de4ba9c-624a-4a12-9957-2778c9983213" containerID="acdbe1114d7c43f56e0402d45e878bd71a840d89cf031ae36d88bfabf3afd951" exitCode=0 Nov 01 22:33:18.127091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.124960 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" event={"ID":"7de4ba9c-624a-4a12-9957-2778c9983213","Type":"ContainerDied","Data":"acdbe1114d7c43f56e0402d45e878bd71a840d89cf031ae36d88bfabf3afd951"} Nov 01 22:33:18.549187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.549153 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg"] Nov 01 22:33:18.552844 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.552828 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.564338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.562125 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg"] Nov 01 22:33:18.583306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.583281 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zb8\" (UniqueName: \"kubernetes.io/projected/bee1eeb2-657c-40cd-b639-d151d93ca14b-kube-api-access-46zb8\") pod \"maas-api-revoke-sub-40b140f2-j7klg\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.583467 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.583326 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40b140f2-j7klg\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.583528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.583481 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40b140f2-j7klg\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.684518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.684491 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40b140f2-j7klg\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.684689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.684627 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zb8\" (UniqueName: \"kubernetes.io/projected/bee1eeb2-657c-40cd-b639-d151d93ca14b-kube-api-access-46zb8\") pod \"maas-api-revoke-sub-40b140f2-j7klg\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.684689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.684681 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40b140f2-j7klg\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.685301 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.685276 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40b140f2-j7klg\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.686978 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.686956 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40b140f2-j7klg\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.695458 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.695433 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zb8\" (UniqueName: \"kubernetes.io/projected/bee1eeb2-657c-40cd-b639-d151d93ca14b-kube-api-access-46zb8\") pod \"maas-api-revoke-sub-40b140f2-j7klg\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:18.872691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:18.872617 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:19.148673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.148640 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg"] Nov 01 22:33:19.149746 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:19.149718 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee1eeb2_657c_40cd_b639_d151d93ca14b.slice/crio-425b203621b6e42e98377e8ecacea094cd0a7595faa043cf86fd3552ca1e7aaa WatchSource:0}: Error finding container 425b203621b6e42e98377e8ecacea094cd0a7595faa043cf86fd3552ca1e7aaa: Status 404 returned error can't find the container with id 425b203621b6e42e98377e8ecacea094cd0a7595faa043cf86fd3552ca1e7aaa Nov 01 22:33:19.268795 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.268767 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:19.292464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.292424 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-kube-api-access-d9v8h\" (UniqueName: \"kubernetes.io/projected/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-kube-api-access-d9v8h\") pod \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " Nov 01 22:33:19.292623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.292511 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-openshift-ca\") pod \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " Nov 01 22:33:19.292685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.292646 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-xks-ca\") pod \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\" (UID: \"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc\") " Nov 01 22:33:19.293811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.293780 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-openshift-ca" pod "378cb8d5-70b2-4dd3-aed1-61fc626cd5cc" (UID: "378cb8d5-70b2-4dd3-aed1-61fc626cd5cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:19.296366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.296338 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-kube-api-access-d9v8h" pod "378cb8d5-70b2-4dd3-aed1-61fc626cd5cc" (UID: "378cb8d5-70b2-4dd3-aed1-61fc626cd5cc"). InnerVolumeSpecName "kube-api-access-d9v8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:19.296476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.296420 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-xks-ca" pod "378cb8d5-70b2-4dd3-aed1-61fc626cd5cc" (UID: "378cb8d5-70b2-4dd3-aed1-61fc626cd5cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:19.320245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.320219 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:19.393723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.393693 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de4ba9c-624a-4a12-9957-2778c9983213-kube-api-access-blhkk\" (UniqueName: \"kubernetes.io/projected/7de4ba9c-624a-4a12-9957-2778c9983213-kube-api-access-blhkk\") pod \"7de4ba9c-624a-4a12-9957-2778c9983213\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " Nov 01 22:33:19.393870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.393760 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-xks-ca\") pod \"7de4ba9c-624a-4a12-9957-2778c9983213\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " Nov 01 22:33:19.393937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.393907 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-openshift-ca\") pod \"7de4ba9c-624a-4a12-9957-2778c9983213\" (UID: \"7de4ba9c-624a-4a12-9957-2778c9983213\") " Nov 01 22:33:19.394255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.394229 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-openshift-ca" pod "7de4ba9c-624a-4a12-9957-2778c9983213" (UID: "7de4ba9c-624a-4a12-9957-2778c9983213"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:19.394384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.394363 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:19.394449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.394386 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9v8h\" (UniqueName: \"kubernetes.io/projected/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-kube-api-access-d9v8h\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:19.394449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.394401 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:19.394449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.394416 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:19.395740 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.395713 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-xks-ca" pod "7de4ba9c-624a-4a12-9957-2778c9983213" (UID: "7de4ba9c-624a-4a12-9957-2778c9983213"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:19.395845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.395828 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de4ba9c-624a-4a12-9957-2778c9983213-kube-api-access-blhkk" pod "7de4ba9c-624a-4a12-9957-2778c9983213" (UID: "7de4ba9c-624a-4a12-9957-2778c9983213"). InnerVolumeSpecName "kube-api-access-blhkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:19.494990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.494965 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhkk\" (UniqueName: \"kubernetes.io/projected/7de4ba9c-624a-4a12-9957-2778c9983213-kube-api-access-blhkk\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:19.494990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:19.494989 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de4ba9c-624a-4a12-9957-2778c9983213-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:20.141898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:20.141860 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" event={"ID":"378cb8d5-70b2-4dd3-aed1-61fc626cd5cc","Type":"ContainerDied","Data":"c7d94b93b8e7a188d14c631aa7baa4143616b4450e860f7cf85ded67d4b7bf0f"} Nov 01 22:33:20.141898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:20.141880 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n" Nov 01 22:33:20.142182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:20.141916 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7d94b93b8e7a188d14c631aa7baa4143616b4450e860f7cf85ded67d4b7bf0f" Nov 01 22:33:20.144751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:20.144723 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" event={"ID":"bee1eeb2-657c-40cd-b639-d151d93ca14b","Type":"ContainerStarted","Data":"07c30c610a77e09c10576a2b9ea76dba54ba3978516a5362442ff16daa8ca230"} Nov 01 22:33:20.145822 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:20.145788 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" event={"ID":"bee1eeb2-657c-40cd-b639-d151d93ca14b","Type":"ContainerStarted","Data":"425b203621b6e42e98377e8ecacea094cd0a7595faa043cf86fd3552ca1e7aaa"} Nov 01 22:33:20.146787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:20.146766 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" event={"ID":"7de4ba9c-624a-4a12-9957-2778c9983213","Type":"ContainerDied","Data":"3e406a1dde18df1aa8a9f843ccb4f89001b8510b169722ec603bae91f9ee4587"} Nov 01 22:33:20.146787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:20.146781 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x" Nov 01 22:33:20.146959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:20.146799 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e406a1dde18df1aa8a9f843ccb4f89001b8510b169722ec603bae91f9ee4587" Nov 01 22:33:20.169264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:20.169221 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" podStartSLOduration=2.1692082839999998 podStartE2EDuration="2.169208284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:20.167417392 +0000 UTC m=+612.815916710" watchObservedRunningTime="2026-11-01 22:33:20.169208284 +0000 UTC m=+612.817707518" Nov 01 22:33:21.154211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.154182 6287 generic.go:356] "Generic (PLEG): container finished" podID="bee1eeb2-657c-40cd-b639-d151d93ca14b" containerID="07c30c610a77e09c10576a2b9ea76dba54ba3978516a5362442ff16daa8ca230" exitCode=0 Nov 01 22:33:21.154403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.154242 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" event={"ID":"bee1eeb2-657c-40cd-b639-d151d93ca14b","Type":"ContainerDied","Data":"07c30c610a77e09c10576a2b9ea76dba54ba3978516a5362442ff16daa8ca230"} Nov 01 22:33:21.888635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.888598 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7"] Nov 01 22:33:21.889091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.889078 6287 cpu_manager.go:414] "containerMap: removing container" podUID="378cb8d5-70b2-4dd3-aed1-61fc626cd5cc" containerName="revoke-keys" Nov 01 22:33:21.889151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.889091 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378cb8d5-70b2-4dd3-aed1-61fc626cd5cc" containerName="revoke-keys" Nov 01 22:33:21.889151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.889102 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7de4ba9c-624a-4a12-9957-2778c9983213" containerName="revoke-keys" Nov 01 22:33:21.889151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.889107 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de4ba9c-624a-4a12-9957-2778c9983213" containerName="revoke-keys" Nov 01 22:33:21.889244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.889217 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="378cb8d5-70b2-4dd3-aed1-61fc626cd5cc" containerName="revoke-keys" Nov 01 22:33:21.889244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.889227 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de4ba9c-624a-4a12-9957-2778c9983213" containerName="revoke-keys" Nov 01 22:33:21.892525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.892509 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:21.900891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.900865 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7"] Nov 01 22:33:21.930773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.930744 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttqwr\" (UniqueName: \"kubernetes.io/projected/f76790c6-2608-495a-868b-602a737cb506-kube-api-access-ttqwr\") pod \"maas-api-revoke-sub-5b59beec-jrjg7\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:21.930915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.930817 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76790c6-2608-495a-868b-602a737cb506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b59beec-jrjg7\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:21.930915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:21.930891 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76790c6-2608-495a-868b-602a737cb506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b59beec-jrjg7\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:22.031745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.031716 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76790c6-2608-495a-868b-602a737cb506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b59beec-jrjg7\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:22.031884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.031765 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76790c6-2608-495a-868b-602a737cb506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b59beec-jrjg7\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:22.031884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.031849 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttqwr\" (UniqueName: \"kubernetes.io/projected/f76790c6-2608-495a-868b-602a737cb506-kube-api-access-ttqwr\") pod \"maas-api-revoke-sub-5b59beec-jrjg7\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:22.032449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.032425 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76790c6-2608-495a-868b-602a737cb506-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b59beec-jrjg7\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:22.034043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.034018 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76790c6-2608-495a-868b-602a737cb506-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b59beec-jrjg7\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:22.039185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.039165 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttqwr\" (UniqueName: \"kubernetes.io/projected/f76790c6-2608-495a-868b-602a737cb506-kube-api-access-ttqwr\") pod \"maas-api-revoke-sub-5b59beec-jrjg7\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:22.210181 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.210153 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:22.303414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.303384 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:22.335751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.335724 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-xks-ca\") pod \"bee1eeb2-657c-40cd-b639-d151d93ca14b\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " Nov 01 22:33:22.335902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.335811 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee1eeb2-657c-40cd-b639-d151d93ca14b-kube-api-access-46zb8\" (UniqueName: \"kubernetes.io/projected/bee1eeb2-657c-40cd-b639-d151d93ca14b-kube-api-access-46zb8\") pod \"bee1eeb2-657c-40cd-b639-d151d93ca14b\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " Nov 01 22:33:22.335983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.335910 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-openshift-ca\") pod \"bee1eeb2-657c-40cd-b639-d151d93ca14b\" (UID: \"bee1eeb2-657c-40cd-b639-d151d93ca14b\") " Nov 01 22:33:22.336342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.336312 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-openshift-ca" pod "bee1eeb2-657c-40cd-b639-d151d93ca14b" (UID: "bee1eeb2-657c-40cd-b639-d151d93ca14b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:22.336471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.336391 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:22.338389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.338363 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-xks-ca" pod "bee1eeb2-657c-40cd-b639-d151d93ca14b" (UID: "bee1eeb2-657c-40cd-b639-d151d93ca14b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:22.338525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.338500 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee1eeb2-657c-40cd-b639-d151d93ca14b-kube-api-access-46zb8" pod "bee1eeb2-657c-40cd-b639-d151d93ca14b" (UID: "bee1eeb2-657c-40cd-b639-d151d93ca14b"). InnerVolumeSpecName "kube-api-access-46zb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:22.437700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.437622 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zb8\" (UniqueName: \"kubernetes.io/projected/bee1eeb2-657c-40cd-b639-d151d93ca14b-kube-api-access-46zb8\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:22.437700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.437650 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bee1eeb2-657c-40cd-b639-d151d93ca14b-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:22.505495 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:22.505470 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7"] Nov 01 22:33:22.506317 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:22.506292 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76790c6_2608_495a_868b_602a737cb506.slice/crio-031537a48ffa74a1b5d2425da7ba92c65418705a6c535d25c594594fd5116161 WatchSource:0}: Error finding container 031537a48ffa74a1b5d2425da7ba92c65418705a6c535d25c594594fd5116161: Status 404 returned error can't find the container with id 031537a48ffa74a1b5d2425da7ba92c65418705a6c535d25c594594fd5116161 Nov 01 22:33:23.168442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:23.168411 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" Nov 01 22:33:23.168442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:23.168437 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg" event={"ID":"bee1eeb2-657c-40cd-b639-d151d93ca14b","Type":"ContainerDied","Data":"425b203621b6e42e98377e8ecacea094cd0a7595faa043cf86fd3552ca1e7aaa"} Nov 01 22:33:23.168921 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:23.168486 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="425b203621b6e42e98377e8ecacea094cd0a7595faa043cf86fd3552ca1e7aaa" Nov 01 22:33:23.170265 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:23.170240 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" event={"ID":"f76790c6-2608-495a-868b-602a737cb506","Type":"ContainerStarted","Data":"3ed4580f398e75b90f82817d2768372a942d170a3b44d3426f6ec6e611f4623e"} Nov 01 22:33:23.170381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:23.170282 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" event={"ID":"f76790c6-2608-495a-868b-602a737cb506","Type":"ContainerStarted","Data":"031537a48ffa74a1b5d2425da7ba92c65418705a6c535d25c594594fd5116161"} Nov 01 22:33:23.191192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:23.191144 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" podStartSLOduration=2.191130113 podStartE2EDuration="2.191130113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:23.18729721 +0000 UTC m=+615.835796447" watchObservedRunningTime="2026-11-01 22:33:23.191130113 +0000 UTC m=+615.839629350" Nov 01 22:33:24.112190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.112159 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j"] Nov 01 22:33:24.112715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.112701 6287 cpu_manager.go:414] "containerMap: removing container" podUID="bee1eeb2-657c-40cd-b639-d151d93ca14b" containerName="revoke-keys" Nov 01 22:33:24.112775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.112717 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee1eeb2-657c-40cd-b639-d151d93ca14b" containerName="revoke-keys" Nov 01 22:33:24.112841 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.112831 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee1eeb2-657c-40cd-b639-d151d93ca14b" containerName="revoke-keys" Nov 01 22:33:24.116240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.116224 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.124738 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.124710 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j"] Nov 01 22:33:24.155949 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.155925 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdls\" (UniqueName: \"kubernetes.io/projected/567de28c-cc74-474f-9503-fb7b06315651-kube-api-access-9zdls\") pod \"maas-api-revoke-sub-2d8da5e5-czh9j\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.156091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.155974 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567de28c-cc74-474f-9503-fb7b06315651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d8da5e5-czh9j\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.156219 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.156197 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567de28c-cc74-474f-9503-fb7b06315651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d8da5e5-czh9j\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.177908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.177884 6287 generic.go:356] "Generic (PLEG): container finished" podID="f76790c6-2608-495a-868b-602a737cb506" containerID="3ed4580f398e75b90f82817d2768372a942d170a3b44d3426f6ec6e611f4623e" exitCode=0 Nov 01 22:33:24.178245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.177972 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" event={"ID":"f76790c6-2608-495a-868b-602a737cb506","Type":"ContainerDied","Data":"3ed4580f398e75b90f82817d2768372a942d170a3b44d3426f6ec6e611f4623e"} Nov 01 22:33:24.257674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.257646 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567de28c-cc74-474f-9503-fb7b06315651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d8da5e5-czh9j\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.257832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.257794 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdls\" (UniqueName: \"kubernetes.io/projected/567de28c-cc74-474f-9503-fb7b06315651-kube-api-access-9zdls\") pod \"maas-api-revoke-sub-2d8da5e5-czh9j\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.257884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.257854 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567de28c-cc74-474f-9503-fb7b06315651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d8da5e5-czh9j\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.258490 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.258467 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567de28c-cc74-474f-9503-fb7b06315651-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d8da5e5-czh9j\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.260000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.259977 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567de28c-cc74-474f-9503-fb7b06315651-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d8da5e5-czh9j\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.266930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.266906 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdls\" (UniqueName: \"kubernetes.io/projected/567de28c-cc74-474f-9503-fb7b06315651-kube-api-access-9zdls\") pod \"maas-api-revoke-sub-2d8da5e5-czh9j\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.432777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.432699 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:24.717465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:24.717433 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j"] Nov 01 22:33:24.717930 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:24.717896 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567de28c_cc74_474f_9503_fb7b06315651.slice/crio-f0c50a1f2a1de885b2bc15b073d2ee312b1b63d0752c600cee08d01f063eb620 WatchSource:0}: Error finding container f0c50a1f2a1de885b2bc15b073d2ee312b1b63d0752c600cee08d01f063eb620: Status 404 returned error can't find the container with id f0c50a1f2a1de885b2bc15b073d2ee312b1b63d0752c600cee08d01f063eb620 Nov 01 22:33:25.183457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.183426 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx"] Nov 01 22:33:25.187566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.187538 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.187770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.187746 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" event={"ID":"567de28c-cc74-474f-9503-fb7b06315651","Type":"ContainerStarted","Data":"372b7e2ada4246e969ac01b0631960407e2fd274f148d999cc98d802bc6b0eea"} Nov 01 22:33:25.187825 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.187791 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" event={"ID":"567de28c-cc74-474f-9503-fb7b06315651","Type":"ContainerStarted","Data":"f0c50a1f2a1de885b2bc15b073d2ee312b1b63d0752c600cee08d01f063eb620"} Nov 01 22:33:25.195845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.195819 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx"] Nov 01 22:33:25.227151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.224035 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" podStartSLOduration=1.224018043 podStartE2EDuration="1.224018043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:25.222265064 +0000 UTC m=+617.870764300" watchObservedRunningTime="2026-11-01 22:33:25.224018043 +0000 UTC m=+617.872517278" Nov 01 22:33:25.270581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.270378 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe83d41-5rxbx\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.270581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.270497 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe83d41-5rxbx\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.271002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.270976 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smhc\" (UniqueName: \"kubernetes.io/projected/5ce6ee89-77f5-4483-837d-13fa77be10df-kube-api-access-6smhc\") pod \"maas-api-revoke-sub-4fe83d41-5rxbx\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.359671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.359649 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:25.372739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.372707 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe83d41-5rxbx\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.372866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.372774 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smhc\" (UniqueName: \"kubernetes.io/projected/5ce6ee89-77f5-4483-837d-13fa77be10df-kube-api-access-6smhc\") pod \"maas-api-revoke-sub-4fe83d41-5rxbx\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.372933 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.372896 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe83d41-5rxbx\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.373337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.373309 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe83d41-5rxbx\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.376525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.376029 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe83d41-5rxbx\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.390702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.390668 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smhc\" (UniqueName: \"kubernetes.io/projected/5ce6ee89-77f5-4483-837d-13fa77be10df-kube-api-access-6smhc\") pod \"maas-api-revoke-sub-4fe83d41-5rxbx\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.473435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.473407 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76790c6-2608-495a-868b-602a737cb506-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76790c6-2608-495a-868b-602a737cb506-maas-api-xks-ca\") pod \"f76790c6-2608-495a-868b-602a737cb506\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " Nov 01 22:33:25.473615 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.473598 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76790c6-2608-495a-868b-602a737cb506-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76790c6-2608-495a-868b-602a737cb506-maas-api-openshift-ca\") pod \"f76790c6-2608-495a-868b-602a737cb506\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " Nov 01 22:33:25.473678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.473664 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76790c6-2608-495a-868b-602a737cb506-kube-api-access-ttqwr\" (UniqueName: \"kubernetes.io/projected/f76790c6-2608-495a-868b-602a737cb506-kube-api-access-ttqwr\") pod \"f76790c6-2608-495a-868b-602a737cb506\" (UID: \"f76790c6-2608-495a-868b-602a737cb506\") " Nov 01 22:33:25.473983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.473950 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76790c6-2608-495a-868b-602a737cb506-maas-api-openshift-ca" pod "f76790c6-2608-495a-868b-602a737cb506" (UID: "f76790c6-2608-495a-868b-602a737cb506"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:25.474099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.474080 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76790c6-2608-495a-868b-602a737cb506-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:25.475742 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.475717 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76790c6-2608-495a-868b-602a737cb506-kube-api-access-ttqwr" pod "f76790c6-2608-495a-868b-602a737cb506" (UID: "f76790c6-2608-495a-868b-602a737cb506"). InnerVolumeSpecName "kube-api-access-ttqwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:25.475850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.475724 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76790c6-2608-495a-868b-602a737cb506-maas-api-xks-ca" pod "f76790c6-2608-495a-868b-602a737cb506" (UID: "f76790c6-2608-495a-868b-602a737cb506"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:25.510984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.510933 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:25.575574 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.575553 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttqwr\" (UniqueName: \"kubernetes.io/projected/f76790c6-2608-495a-868b-602a737cb506-kube-api-access-ttqwr\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:25.575681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.575581 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76790c6-2608-495a-868b-602a737cb506-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:25.784706 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:25.784676 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx"] Nov 01 22:33:25.785623 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:25.785598 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce6ee89_77f5_4483_837d_13fa77be10df.slice/crio-8518ac28268111347b7ec5ee1e8058f0c7e603735187dc9eb3417537e980ec4a WatchSource:0}: Error finding container 8518ac28268111347b7ec5ee1e8058f0c7e603735187dc9eb3417537e980ec4a: Status 404 returned error can't find the container with id 8518ac28268111347b7ec5ee1e8058f0c7e603735187dc9eb3417537e980ec4a Nov 01 22:33:26.197262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.196834 6287 generic.go:356] "Generic (PLEG): container finished" podID="567de28c-cc74-474f-9503-fb7b06315651" containerID="372b7e2ada4246e969ac01b0631960407e2fd274f148d999cc98d802bc6b0eea" exitCode=0 Nov 01 22:33:26.197262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.197063 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" event={"ID":"567de28c-cc74-474f-9503-fb7b06315651","Type":"ContainerDied","Data":"372b7e2ada4246e969ac01b0631960407e2fd274f148d999cc98d802bc6b0eea"} Nov 01 22:33:26.198677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.198656 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" Nov 01 22:33:26.198677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.198665 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7" event={"ID":"f76790c6-2608-495a-868b-602a737cb506","Type":"ContainerDied","Data":"031537a48ffa74a1b5d2425da7ba92c65418705a6c535d25c594594fd5116161"} Nov 01 22:33:26.198854 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.198713 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="031537a48ffa74a1b5d2425da7ba92c65418705a6c535d25c594594fd5116161" Nov 01 22:33:26.200631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.200603 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" event={"ID":"5ce6ee89-77f5-4483-837d-13fa77be10df","Type":"ContainerStarted","Data":"95481a4779806e911abf76f276baefdf70dd5ee410abc151d6f800c91664d52c"} Nov 01 22:33:26.200743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.200654 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" event={"ID":"5ce6ee89-77f5-4483-837d-13fa77be10df","Type":"ContainerStarted","Data":"8518ac28268111347b7ec5ee1e8058f0c7e603735187dc9eb3417537e980ec4a"} Nov 01 22:33:26.200820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.200806 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg"] Nov 01 22:33:26.201382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.201368 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f76790c6-2608-495a-868b-602a737cb506" containerName="revoke-keys" Nov 01 22:33:26.201434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.201382 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76790c6-2608-495a-868b-602a737cb506" containerName="revoke-keys" Nov 01 22:33:26.201492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.201483 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76790c6-2608-495a-868b-602a737cb506" containerName="revoke-keys" Nov 01 22:33:26.204735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.204720 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.211980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.211623 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg"] Nov 01 22:33:26.253423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.253365 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" podStartSLOduration=1.253348489 podStartE2EDuration="1.253348489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:26.249979266 +0000 UTC m=+618.898478500" watchObservedRunningTime="2026-11-01 22:33:26.253348489 +0000 UTC m=+618.901847725" Nov 01 22:33:26.282993 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.282964 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6139e278-lb6wg\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.283149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.283107 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6139e278-lb6wg\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.283328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.283308 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhjvq\" (UniqueName: \"kubernetes.io/projected/5fe46b38-bb6b-4803-ac64-1764daf1a252-kube-api-access-jhjvq\") pod \"maas-api-revoke-sub-6139e278-lb6wg\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.384168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.384093 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6139e278-lb6wg\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.384307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.384205 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6139e278-lb6wg\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.384393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.384355 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhjvq\" (UniqueName: \"kubernetes.io/projected/5fe46b38-bb6b-4803-ac64-1764daf1a252-kube-api-access-jhjvq\") pod \"maas-api-revoke-sub-6139e278-lb6wg\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.384844 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.384818 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6139e278-lb6wg\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.386410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.386392 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6139e278-lb6wg\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.392161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.392139 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhjvq\" (UniqueName: \"kubernetes.io/projected/5fe46b38-bb6b-4803-ac64-1764daf1a252-kube-api-access-jhjvq\") pod \"maas-api-revoke-sub-6139e278-lb6wg\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.523596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.523565 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:26.797793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:26.797766 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg"] Nov 01 22:33:26.798904 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:26.798875 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe46b38_bb6b_4803_ac64_1764daf1a252.slice/crio-e5959819da3a6ae220b1e04830fc0906ac4ee28cab60a2e206efbf9ce78defcc WatchSource:0}: Error finding container e5959819da3a6ae220b1e04830fc0906ac4ee28cab60a2e206efbf9ce78defcc: Status 404 returned error can't find the container with id e5959819da3a6ae220b1e04830fc0906ac4ee28cab60a2e206efbf9ce78defcc Nov 01 22:33:27.209541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.209509 6287 generic.go:356] "Generic (PLEG): container finished" podID="5ce6ee89-77f5-4483-837d-13fa77be10df" containerID="95481a4779806e911abf76f276baefdf70dd5ee410abc151d6f800c91664d52c" exitCode=0 Nov 01 22:33:27.209939 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.209595 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" event={"ID":"5ce6ee89-77f5-4483-837d-13fa77be10df","Type":"ContainerDied","Data":"95481a4779806e911abf76f276baefdf70dd5ee410abc151d6f800c91664d52c"} Nov 01 22:33:27.211423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.211399 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" event={"ID":"5fe46b38-bb6b-4803-ac64-1764daf1a252","Type":"ContainerStarted","Data":"c881e38c3555465ed424b44466e52bdf7a1369247f7654ea6328b3546a4bb2f3"} Nov 01 22:33:27.211549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.211440 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" event={"ID":"5fe46b38-bb6b-4803-ac64-1764daf1a252","Type":"ContainerStarted","Data":"e5959819da3a6ae220b1e04830fc0906ac4ee28cab60a2e206efbf9ce78defcc"} Nov 01 22:33:27.246742 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.246340 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" podStartSLOduration=1.246322472 podStartE2EDuration="1.246322472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:27.245592648 +0000 UTC m=+619.894091883" watchObservedRunningTime="2026-11-01 22:33:27.246322472 +0000 UTC m=+619.894821708" Nov 01 22:33:27.383830 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.383795 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:27.497232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.497161 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/567de28c-cc74-474f-9503-fb7b06315651-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567de28c-cc74-474f-9503-fb7b06315651-maas-api-xks-ca\") pod \"567de28c-cc74-474f-9503-fb7b06315651\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " Nov 01 22:33:27.497375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.497291 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567de28c-cc74-474f-9503-fb7b06315651-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567de28c-cc74-474f-9503-fb7b06315651-maas-api-openshift-ca\") pod \"567de28c-cc74-474f-9503-fb7b06315651\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " Nov 01 22:33:27.497375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.497333 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567de28c-cc74-474f-9503-fb7b06315651-kube-api-access-9zdls\" (UniqueName: \"kubernetes.io/projected/567de28c-cc74-474f-9503-fb7b06315651-kube-api-access-9zdls\") pod \"567de28c-cc74-474f-9503-fb7b06315651\" (UID: \"567de28c-cc74-474f-9503-fb7b06315651\") " Nov 01 22:33:27.497682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.497647 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567de28c-cc74-474f-9503-fb7b06315651-maas-api-openshift-ca" pod "567de28c-cc74-474f-9503-fb7b06315651" (UID: "567de28c-cc74-474f-9503-fb7b06315651"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:27.497831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.497802 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/567de28c-cc74-474f-9503-fb7b06315651-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:27.499097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.499073 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567de28c-cc74-474f-9503-fb7b06315651-maas-api-xks-ca" pod "567de28c-cc74-474f-9503-fb7b06315651" (UID: "567de28c-cc74-474f-9503-fb7b06315651"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:27.499417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.499393 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567de28c-cc74-474f-9503-fb7b06315651-kube-api-access-9zdls" pod "567de28c-cc74-474f-9503-fb7b06315651" (UID: "567de28c-cc74-474f-9503-fb7b06315651"). InnerVolumeSpecName "kube-api-access-9zdls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:27.598733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.598706 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/567de28c-cc74-474f-9503-fb7b06315651-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:27.598733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:27.598730 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zdls\" (UniqueName: \"kubernetes.io/projected/567de28c-cc74-474f-9503-fb7b06315651-kube-api-access-9zdls\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:28.225164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.225122 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" event={"ID":"567de28c-cc74-474f-9503-fb7b06315651","Type":"ContainerDied","Data":"f0c50a1f2a1de885b2bc15b073d2ee312b1b63d0752c600cee08d01f063eb620"} Nov 01 22:33:28.225164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.225157 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j" Nov 01 22:33:28.225741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.225190 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0c50a1f2a1de885b2bc15b073d2ee312b1b63d0752c600cee08d01f063eb620" Nov 01 22:33:28.228199 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.228161 6287 generic.go:356] "Generic (PLEG): container finished" podID="5fe46b38-bb6b-4803-ac64-1764daf1a252" containerID="c881e38c3555465ed424b44466e52bdf7a1369247f7654ea6328b3546a4bb2f3" exitCode=0 Nov 01 22:33:28.228348 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.228207 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" event={"ID":"5fe46b38-bb6b-4803-ac64-1764daf1a252","Type":"ContainerDied","Data":"c881e38c3555465ed424b44466e52bdf7a1369247f7654ea6328b3546a4bb2f3"} Nov 01 22:33:28.379741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.379709 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:28.512680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.512648 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-openshift-ca\") pod \"5ce6ee89-77f5-4483-837d-13fa77be10df\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " Nov 01 22:33:28.512867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.512693 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce6ee89-77f5-4483-837d-13fa77be10df-kube-api-access-6smhc\" (UniqueName: \"kubernetes.io/projected/5ce6ee89-77f5-4483-837d-13fa77be10df-kube-api-access-6smhc\") pod \"5ce6ee89-77f5-4483-837d-13fa77be10df\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " Nov 01 22:33:28.512867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.512772 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-xks-ca\") pod \"5ce6ee89-77f5-4483-837d-13fa77be10df\" (UID: \"5ce6ee89-77f5-4483-837d-13fa77be10df\") " Nov 01 22:33:28.513072 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.513042 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-openshift-ca" pod "5ce6ee89-77f5-4483-837d-13fa77be10df" (UID: "5ce6ee89-77f5-4483-837d-13fa77be10df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:28.513352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.513329 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:28.514772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.514751 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce6ee89-77f5-4483-837d-13fa77be10df-kube-api-access-6smhc" pod "5ce6ee89-77f5-4483-837d-13fa77be10df" (UID: "5ce6ee89-77f5-4483-837d-13fa77be10df"). InnerVolumeSpecName "kube-api-access-6smhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:28.514862 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.514751 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-xks-ca" pod "5ce6ee89-77f5-4483-837d-13fa77be10df" (UID: "5ce6ee89-77f5-4483-837d-13fa77be10df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:28.615136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.614626 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6smhc\" (UniqueName: \"kubernetes.io/projected/5ce6ee89-77f5-4483-837d-13fa77be10df-kube-api-access-6smhc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:28.615136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.614663 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce6ee89-77f5-4483-837d-13fa77be10df-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:28.783599 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.783506 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt"] Nov 01 22:33:28.784393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.784366 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5ce6ee89-77f5-4483-837d-13fa77be10df" containerName="revoke-keys" Nov 01 22:33:28.784393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.784392 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce6ee89-77f5-4483-837d-13fa77be10df" containerName="revoke-keys" Nov 01 22:33:28.784562 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.784420 6287 cpu_manager.go:414] "containerMap: removing container" podUID="567de28c-cc74-474f-9503-fb7b06315651" containerName="revoke-keys" Nov 01 22:33:28.784562 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.784429 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567de28c-cc74-474f-9503-fb7b06315651" containerName="revoke-keys" Nov 01 22:33:28.784654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.784569 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="567de28c-cc74-474f-9503-fb7b06315651" containerName="revoke-keys" Nov 01 22:33:28.784654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.784599 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce6ee89-77f5-4483-837d-13fa77be10df" containerName="revoke-keys" Nov 01 22:33:28.788623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.788602 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:28.793201 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.793173 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b42cc4-openshift-default-dockercfg-lt2tw\"" Nov 01 22:33:28.803601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.803577 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt"] Nov 01 22:33:28.839027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.838996 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh"] Nov 01 22:33:28.843852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.843834 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:28.849658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.849634 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh"] Nov 01 22:33:28.917856 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.917826 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtrc\" (UniqueName: \"kubernetes.io/projected/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-kube-api-access-pmtrc\") pod \"maas-api-revoke-sub-17d2e78f-cdmnh\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:28.918019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.917869 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:28.918019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.917889 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:28.918019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.917976 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17d2e78f-cdmnh\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:28.918205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.918062 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:28.918205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.918085 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:28.918205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.918123 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:28.918316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.918233 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:28.918316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.918279 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:28.918397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.918323 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:28.918397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.918363 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17d2e78f-cdmnh\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:28.918397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:28.918391 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqkz\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-kube-api-access-stqkz\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.019646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019616 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17d2e78f-cdmnh\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:29.019793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019654 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqkz\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-kube-api-access-stqkz\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.019793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019706 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtrc\" (UniqueName: \"kubernetes.io/projected/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-kube-api-access-pmtrc\") pod \"maas-api-revoke-sub-17d2e78f-cdmnh\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:29.019793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019731 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.019793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019750 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.019793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019778 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17d2e78f-cdmnh\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:29.020040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019832 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019854 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019880 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.019945 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.020002 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.020036 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020388 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.020154 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020388 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.020195 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.020424 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.020624 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17d2e78f-cdmnh\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:29.020715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.020645 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.020766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.020750 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.022254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.022234 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17d2e78f-cdmnh\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:29.022589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.022568 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.022721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.022705 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.027549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.027526 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.027635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.027615 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqkz\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-kube-api-access-stqkz\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.028035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.028019 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtrc\" (UniqueName: \"kubernetes.io/projected/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-kube-api-access-pmtrc\") pod \"maas-api-revoke-sub-17d2e78f-cdmnh\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:29.104602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.104548 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:29.164327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.162683 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:29.249155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.244861 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" Nov 01 22:33:29.249155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.246202 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx" event={"ID":"5ce6ee89-77f5-4483-837d-13fa77be10df","Type":"ContainerDied","Data":"8518ac28268111347b7ec5ee1e8058f0c7e603735187dc9eb3417537e980ec4a"} Nov 01 22:33:29.249155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.246253 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8518ac28268111347b7ec5ee1e8058f0c7e603735187dc9eb3417537e980ec4a" Nov 01 22:33:29.393734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.393703 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt"] Nov 01 22:33:29.396925 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:29.396898 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980ff60d_a1fc_4f07_82ff_f916b4686bc5.slice/crio-62021e6a1d037d23cdde1b81ad6332dec840f5558444e486312fcf38afe9108c WatchSource:0}: Error finding container 62021e6a1d037d23cdde1b81ad6332dec840f5558444e486312fcf38afe9108c: Status 404 returned error can't find the container with id 62021e6a1d037d23cdde1b81ad6332dec840f5558444e486312fcf38afe9108c Nov 01 22:33:29.403376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.403336 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:33:29.403469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.403418 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:33:29.403469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.403461 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:33:29.423605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.423584 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:29.479904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.479881 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh"] Nov 01 22:33:29.526759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.526724 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-xks-ca\") pod \"5fe46b38-bb6b-4803-ac64-1764daf1a252\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " Nov 01 22:33:29.526930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.526790 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe46b38-bb6b-4803-ac64-1764daf1a252-kube-api-access-jhjvq\" (UniqueName: \"kubernetes.io/projected/5fe46b38-bb6b-4803-ac64-1764daf1a252-kube-api-access-jhjvq\") pod \"5fe46b38-bb6b-4803-ac64-1764daf1a252\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " Nov 01 22:33:29.526930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.526894 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-openshift-ca\") pod \"5fe46b38-bb6b-4803-ac64-1764daf1a252\" (UID: \"5fe46b38-bb6b-4803-ac64-1764daf1a252\") " Nov 01 22:33:29.527257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.527224 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-openshift-ca" pod "5fe46b38-bb6b-4803-ac64-1764daf1a252" (UID: "5fe46b38-bb6b-4803-ac64-1764daf1a252"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:29.527485 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.527465 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:29.528960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.528930 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe46b38-bb6b-4803-ac64-1764daf1a252-kube-api-access-jhjvq" pod "5fe46b38-bb6b-4803-ac64-1764daf1a252" (UID: "5fe46b38-bb6b-4803-ac64-1764daf1a252"). InnerVolumeSpecName "kube-api-access-jhjvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:29.529043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.528955 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-xks-ca" pod "5fe46b38-bb6b-4803-ac64-1764daf1a252" (UID: "5fe46b38-bb6b-4803-ac64-1764daf1a252"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:29.629142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.629092 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fe46b38-bb6b-4803-ac64-1764daf1a252-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:29.629142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:29.629142 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhjvq\" (UniqueName: \"kubernetes.io/projected/5fe46b38-bb6b-4803-ac64-1764daf1a252-kube-api-access-jhjvq\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:30.254290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:30.254257 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" Nov 01 22:33:30.254290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:30.254269 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg" event={"ID":"5fe46b38-bb6b-4803-ac64-1764daf1a252","Type":"ContainerDied","Data":"e5959819da3a6ae220b1e04830fc0906ac4ee28cab60a2e206efbf9ce78defcc"} Nov 01 22:33:30.254798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:30.254317 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5959819da3a6ae220b1e04830fc0906ac4ee28cab60a2e206efbf9ce78defcc" Nov 01 22:33:30.255977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:30.255950 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" event={"ID":"e4d9fa6f-0b29-4e32-9206-4e267d7bf451","Type":"ContainerStarted","Data":"863b3376d2d1119414737f73feb6d0ab88abfa61d5142ba3e8b40589e8455fce"} Nov 01 22:33:30.256108 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:30.255991 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" event={"ID":"e4d9fa6f-0b29-4e32-9206-4e267d7bf451","Type":"ContainerStarted","Data":"fcc866425818d046f3f353d0dd3e4c1bf5b47f7b5d35aa04db3f3d2d93c16d0a"} Nov 01 22:33:30.257679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:30.257657 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" event={"ID":"980ff60d-a1fc-4f07-82ff-f916b4686bc5","Type":"ContainerStarted","Data":"5f28d40342eb553fcfef13e06da41ca2614e548079b8a64c71e067e1f27ee29a"} Nov 01 22:33:30.257821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:30.257689 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" event={"ID":"980ff60d-a1fc-4f07-82ff-f916b4686bc5","Type":"ContainerStarted","Data":"62021e6a1d037d23cdde1b81ad6332dec840f5558444e486312fcf38afe9108c"} Nov 01 22:33:30.298782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:30.298721 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" podStartSLOduration=2.298702672 podStartE2EDuration="2.298702672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:30.296603417 +0000 UTC m=+622.945102652" watchObservedRunningTime="2026-11-01 22:33:30.298702672 +0000 UTC m=+622.947201909" Nov 01 22:33:30.319943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:30.319883 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" podStartSLOduration=2.319864918 podStartE2EDuration="2.319864918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:30.317578986 +0000 UTC m=+622.966078216" watchObservedRunningTime="2026-11-01 22:33:30.319864918 +0000 UTC m=+622.968364155" Nov 01 22:33:31.106138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:31.105401 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:31.110136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:31.107701 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:33:31.110136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:31.107784 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:33:31.309141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:31.306767 6287 generic.go:356] "Generic (PLEG): container finished" podID="e4d9fa6f-0b29-4e32-9206-4e267d7bf451" containerID="863b3376d2d1119414737f73feb6d0ab88abfa61d5142ba3e8b40589e8455fce" exitCode=0 Nov 01 22:33:31.309141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:31.308021 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" event={"ID":"e4d9fa6f-0b29-4e32-9206-4e267d7bf451","Type":"ContainerDied","Data":"863b3376d2d1119414737f73feb6d0ab88abfa61d5142ba3e8b40589e8455fce"} Nov 01 22:33:32.107136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:32.106289 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:33:32.107136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:32.106354 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:33:32.860298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:32.860275 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:32.969787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:32.969751 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-kube-api-access-pmtrc\" (UniqueName: \"kubernetes.io/projected/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-kube-api-access-pmtrc\") pod \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " Nov 01 22:33:32.970060 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:32.970035 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-xks-ca\") pod \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " Nov 01 22:33:32.992157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:32.991476 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-openshift-ca" pod "e4d9fa6f-0b29-4e32-9206-4e267d7bf451" (UID: "e4d9fa6f-0b29-4e32-9206-4e267d7bf451"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:33.013135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:32.992387 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-openshift-ca\") pod \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\" (UID: \"e4d9fa6f-0b29-4e32-9206-4e267d7bf451\") " Nov 01 22:33:33.026330 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.025922 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-kube-api-access-pmtrc" pod "e4d9fa6f-0b29-4e32-9206-4e267d7bf451" (UID: "e4d9fa6f-0b29-4e32-9206-4e267d7bf451"). InnerVolumeSpecName "kube-api-access-pmtrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:33.056496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.056325 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-xks-ca" pod "e4d9fa6f-0b29-4e32-9206-4e267d7bf451" (UID: "e4d9fa6f-0b29-4e32-9206-4e267d7bf451"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:33.107046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.106900 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmtrc\" (UniqueName: \"kubernetes.io/projected/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-kube-api-access-pmtrc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:33.107046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.106944 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:33.107046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.106959 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9fa6f-0b29-4e32-9206-4e267d7bf451-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:33.115303 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.115272 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:33:33.115547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.115520 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:33:33.346821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.346776 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" event={"ID":"e4d9fa6f-0b29-4e32-9206-4e267d7bf451","Type":"ContainerDied","Data":"fcc866425818d046f3f353d0dd3e4c1bf5b47f7b5d35aa04db3f3d2d93c16d0a"} Nov 01 22:33:33.347034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.346846 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcc866425818d046f3f353d0dd3e4c1bf5b47f7b5d35aa04db3f3d2d93c16d0a" Nov 01 22:33:33.347034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:33.346899 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh" Nov 01 22:33:34.108335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:34.107579 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:33:34.108335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:34.107643 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:33:35.105863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:35.105829 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:33:35.106043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:35.105883 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:33:36.105663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.105626 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:33:36.106067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.105680 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:33:36.825020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.824978 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8"] Nov 01 22:33:36.825559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.825540 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e4d9fa6f-0b29-4e32-9206-4e267d7bf451" containerName="revoke-keys" Nov 01 22:33:36.825651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.825561 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d9fa6f-0b29-4e32-9206-4e267d7bf451" containerName="revoke-keys" Nov 01 22:33:36.825651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.825596 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5fe46b38-bb6b-4803-ac64-1764daf1a252" containerName="revoke-keys" Nov 01 22:33:36.825651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.825604 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe46b38-bb6b-4803-ac64-1764daf1a252" containerName="revoke-keys" Nov 01 22:33:36.825809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.825744 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe46b38-bb6b-4803-ac64-1764daf1a252" containerName="revoke-keys" Nov 01 22:33:36.825809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.825763 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d9fa6f-0b29-4e32-9206-4e267d7bf451" containerName="revoke-keys" Nov 01 22:33:36.829761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.829742 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:36.838679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.838639 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:33:36.839596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.839579 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:33:36.839692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.839665 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:33:36.839746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.839711 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:33:36.845465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.845431 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8"] Nov 01 22:33:36.911323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.910280 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbw6w\" (UniqueName: \"kubernetes.io/projected/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-kube-api-access-kbw6w\") pod \"maas-api-revoke-sub-ca2f2259-f7hq8\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:36.911323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.910365 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca2f2259-f7hq8\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:36.911323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:36.910417 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca2f2259-f7hq8\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:37.012200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.012149 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca2f2259-f7hq8\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:37.012400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.012306 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca2f2259-f7hq8\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:37.012544 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.012520 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbw6w\" (UniqueName: \"kubernetes.io/projected/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-kube-api-access-kbw6w\") pod \"maas-api-revoke-sub-ca2f2259-f7hq8\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:37.013081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.013054 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca2f2259-f7hq8\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:37.016772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.016749 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca2f2259-f7hq8\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:37.025804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.025779 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbw6w\" (UniqueName: \"kubernetes.io/projected/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-kube-api-access-kbw6w\") pod \"maas-api-revoke-sub-ca2f2259-f7hq8\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:37.144012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.143928 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:37.481177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.481154 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8"] Nov 01 22:33:37.591023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.590984 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:37.595135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.592187 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:37.596241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:37.596219 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:33:38.053302 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.053258 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt"] Nov 01 22:33:38.058517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.058451 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.065893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.065856 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt"] Nov 01 22:33:38.258134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.257808 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe75dc32-kspzt\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.258134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.257894 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe75dc32-kspzt\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.258134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.257947 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6bq\" (UniqueName: \"kubernetes.io/projected/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-kube-api-access-9j6bq\") pod \"maas-api-revoke-sub-fe75dc32-kspzt\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.363245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.358836 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe75dc32-kspzt\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.363245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.358951 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe75dc32-kspzt\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.363245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.359011 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6bq\" (UniqueName: \"kubernetes.io/projected/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-kube-api-access-9j6bq\") pod \"maas-api-revoke-sub-fe75dc32-kspzt\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.363245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.360977 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe75dc32-kspzt\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.387411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.386687 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe75dc32-kspzt\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.472058 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.471965 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" event={"ID":"fc8534fc-3426-4c34-bc85-dc1dc63e5d06","Type":"ContainerStarted","Data":"e0ebe3afc7fbff9e6fe0a08e25cf5a9818e673ae34dca00a0c601d990f916b53"} Nov 01 22:33:38.491279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.491246 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" event={"ID":"fc8534fc-3426-4c34-bc85-dc1dc63e5d06","Type":"ContainerStarted","Data":"6b037143c110a7c6f421cc7ec48d4fbcec32d1d10cf72df296f30b6fae83bf32"} Nov 01 22:33:38.514909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.493797 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6bq\" (UniqueName: \"kubernetes.io/projected/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-kube-api-access-9j6bq\") pod \"maas-api-revoke-sub-fe75dc32-kspzt\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:38.525135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.524248 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" podStartSLOduration=2.524229141 podStartE2EDuration="2.524229141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:38.523858825 +0000 UTC m=+631.172358072" watchObservedRunningTime="2026-11-01 22:33:38.524229141 +0000 UTC m=+631.172728378" Nov 01 22:33:38.599332 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:38.596972 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:39.057389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.057302 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt"] Nov 01 22:33:39.061533 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.061512 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.068345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.068316 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt"] Nov 01 22:33:39.096714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.096687 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt"] Nov 01 22:33:39.153786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.153753 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wpc\" (UniqueName: \"kubernetes.io/projected/23a442ac-8dac-4589-be51-54cd2265aa5e-kube-api-access-s6wpc\") pod \"maas-api-revoke-sub-04a27e87-7nlkt\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.153908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.153802 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a27e87-7nlkt\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.153996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.153970 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a27e87-7nlkt\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.255617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.255579 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wpc\" (UniqueName: \"kubernetes.io/projected/23a442ac-8dac-4589-be51-54cd2265aa5e-kube-api-access-s6wpc\") pod \"maas-api-revoke-sub-04a27e87-7nlkt\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.255767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.255639 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a27e87-7nlkt\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.255767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.255743 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a27e87-7nlkt\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.256402 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.256371 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a27e87-7nlkt\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.258052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.258029 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a27e87-7nlkt\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.265703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.265683 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wpc\" (UniqueName: \"kubernetes.io/projected/23a442ac-8dac-4589-be51-54cd2265aa5e-kube-api-access-s6wpc\") pod \"maas-api-revoke-sub-04a27e87-7nlkt\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.391850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.391824 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:39.482516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.482485 6287 generic.go:356] "Generic (PLEG): container finished" podID="fc8534fc-3426-4c34-bc85-dc1dc63e5d06" containerID="e0ebe3afc7fbff9e6fe0a08e25cf5a9818e673ae34dca00a0c601d990f916b53" exitCode=0 Nov 01 22:33:39.482672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.482641 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" event={"ID":"fc8534fc-3426-4c34-bc85-dc1dc63e5d06","Type":"ContainerDied","Data":"e0ebe3afc7fbff9e6fe0a08e25cf5a9818e673ae34dca00a0c601d990f916b53"} Nov 01 22:33:39.485190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.485167 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" event={"ID":"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4","Type":"ContainerStarted","Data":"71f29b856afd2a40ae6bd68391a24e1a9a1325b6ec64edf473e36a74f0159d47"} Nov 01 22:33:39.485424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.485397 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" event={"ID":"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4","Type":"ContainerStarted","Data":"8fac8da1806dfbf4dd85df1b71a1c7b8589e8acb34b7b6665edf8d2037d79787"} Nov 01 22:33:39.522493 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.522437 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" podStartSLOduration=1.522419019 podStartE2EDuration="1.522419019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:39.51938559 +0000 UTC m=+632.167884827" watchObservedRunningTime="2026-11-01 22:33:39.522419019 +0000 UTC m=+632.170918254" Nov 01 22:33:39.615901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.615865 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h"] Nov 01 22:33:39.624569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.624367 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.630217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.630174 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e25182\"/\"distinct-w0-e25182-kserve-self-signed-certs\"" Nov 01 22:33:39.633183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.633101 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h"] Nov 01 22:33:39.661188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.661157 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tmp-dir\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.661370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.661193 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tls-certs\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.661370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.661224 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-home\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.661370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.661275 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-model-cache\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.661370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.661332 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-dshm\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.661370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.661357 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4kl9\" (UniqueName: \"kubernetes.io/projected/5bc29f3d-8367-4958-bb4e-a489d5ab620e-kube-api-access-l4kl9\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.682070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.682044 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt"] Nov 01 22:33:39.684435 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:39.684397 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a442ac_8dac_4589_be51_54cd2265aa5e.slice/crio-0053174bb6cdde05deff2625394f7911452eb9789108cee2ce2576ff56b2556f WatchSource:0}: Error finding container 0053174bb6cdde05deff2625394f7911452eb9789108cee2ce2576ff56b2556f: Status 404 returned error can't find the container with id 0053174bb6cdde05deff2625394f7911452eb9789108cee2ce2576ff56b2556f Nov 01 22:33:39.762383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.762357 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-dshm\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.762522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.762392 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kl9\" (UniqueName: \"kubernetes.io/projected/5bc29f3d-8367-4958-bb4e-a489d5ab620e-kube-api-access-l4kl9\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.762522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.762456 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tmp-dir\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.762522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.762472 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tls-certs\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.762692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.762545 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-home\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.762692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.762616 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-model-cache\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.762988 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.762932 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tmp-dir\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.763047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.763003 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-model-cache\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.763047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.763028 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-home\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.764605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.764583 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-dshm\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.764927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.764910 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tls-certs\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.778902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.778863 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kl9\" (UniqueName: \"kubernetes.io/projected/5bc29f3d-8367-4958-bb4e-a489d5ab620e-kube-api-access-l4kl9\") pod \"distinct-w0-e25182-kserve-96f66848b-b5d2h\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:39.943888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:39.943804 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:40.233009 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:40.232961 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bc29f3d_8367_4958_bb4e_a489d5ab620e.slice/crio-73fc6a92be7b79fe13daef69ac34fdded40165f2923d2219045ff724eb0a1e98 WatchSource:0}: Error finding container 73fc6a92be7b79fe13daef69ac34fdded40165f2923d2219045ff724eb0a1e98: Status 404 returned error can't find the container with id 73fc6a92be7b79fe13daef69ac34fdded40165f2923d2219045ff724eb0a1e98 Nov 01 22:33:40.233327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.233296 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h"] Nov 01 22:33:40.506418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.506372 6287 generic.go:356] "Generic (PLEG): container finished" podID="7b3db5fb-3ee8-4279-8888-14b8e6cc0df4" containerID="71f29b856afd2a40ae6bd68391a24e1a9a1325b6ec64edf473e36a74f0159d47" exitCode=0 Nov 01 22:33:40.506917 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.506508 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" event={"ID":"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4","Type":"ContainerDied","Data":"71f29b856afd2a40ae6bd68391a24e1a9a1325b6ec64edf473e36a74f0159d47"} Nov 01 22:33:40.508749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.508722 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" event={"ID":"5bc29f3d-8367-4958-bb4e-a489d5ab620e","Type":"ContainerStarted","Data":"fb6a47634e7229a6f52025a1044fefc2ececb44cb7ba4b02c20b0d0e384fbb57"} Nov 01 22:33:40.508897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.508878 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" event={"ID":"5bc29f3d-8367-4958-bb4e-a489d5ab620e","Type":"ContainerStarted","Data":"73fc6a92be7b79fe13daef69ac34fdded40165f2923d2219045ff724eb0a1e98"} Nov 01 22:33:40.510674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.510648 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" event={"ID":"23a442ac-8dac-4589-be51-54cd2265aa5e","Type":"ContainerStarted","Data":"ddc97e5497aa7e373793386a2614fc785fba9b6e20563e8ed16a8ddafbbb68fd"} Nov 01 22:33:40.510777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.510692 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" event={"ID":"23a442ac-8dac-4589-be51-54cd2265aa5e","Type":"ContainerStarted","Data":"0053174bb6cdde05deff2625394f7911452eb9789108cee2ce2576ff56b2556f"} Nov 01 22:33:40.543324 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.543254 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podStartSLOduration=1.543237364 podStartE2EDuration="1.543237364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:40.54046018 +0000 UTC m=+633.188959418" watchObservedRunningTime="2026-11-01 22:33:40.543237364 +0000 UTC m=+633.191736599" Nov 01 22:33:40.560215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.560158 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" podStartSLOduration=1.5601390149999999 podStartE2EDuration="1.560139015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:40.556499972 +0000 UTC m=+633.204999207" watchObservedRunningTime="2026-11-01 22:33:40.560139015 +0000 UTC m=+633.208638251" Nov 01 22:33:40.774674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.774592 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:40.902631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.902603 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-xks-ca\") pod \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " Nov 01 22:33:40.902798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.902641 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-kube-api-access-kbw6w\" (UniqueName: \"kubernetes.io/projected/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-kube-api-access-kbw6w\") pod \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " Nov 01 22:33:40.902798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.902742 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-openshift-ca\") pod \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\" (UID: \"fc8534fc-3426-4c34-bc85-dc1dc63e5d06\") " Nov 01 22:33:40.903160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.903123 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-openshift-ca" pod "fc8534fc-3426-4c34-bc85-dc1dc63e5d06" (UID: "fc8534fc-3426-4c34-bc85-dc1dc63e5d06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:40.904728 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.904702 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-xks-ca" pod "fc8534fc-3426-4c34-bc85-dc1dc63e5d06" (UID: "fc8534fc-3426-4c34-bc85-dc1dc63e5d06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:40.904794 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.904723 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-kube-api-access-kbw6w" pod "fc8534fc-3426-4c34-bc85-dc1dc63e5d06" (UID: "fc8534fc-3426-4c34-bc85-dc1dc63e5d06"). InnerVolumeSpecName "kube-api-access-kbw6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:40.943937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.943912 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:40.992714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.992689 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz"] Nov 01 22:33:40.993267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.993249 6287 cpu_manager.go:414] "containerMap: removing container" podUID="fc8534fc-3426-4c34-bc85-dc1dc63e5d06" containerName="revoke-keys" Nov 01 22:33:40.993345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.993268 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8534fc-3426-4c34-bc85-dc1dc63e5d06" containerName="revoke-keys" Nov 01 22:33:40.993425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.993413 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8534fc-3426-4c34-bc85-dc1dc63e5d06" containerName="revoke-keys" Nov 01 22:33:40.996972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:40.996956 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.004238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.004215 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:41.004336 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.004240 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:41.004336 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.004255 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbw6w\" (UniqueName: \"kubernetes.io/projected/fc8534fc-3426-4c34-bc85-dc1dc63e5d06-kube-api-access-kbw6w\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:41.004774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.004751 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz"] Nov 01 22:33:41.104952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.104878 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k99f\" (UniqueName: \"kubernetes.io/projected/0dd50076-b058-41af-8482-638f0fed4601-kube-api-access-6k99f\") pod \"maas-api-revoke-sub-e0ed7ae5-hqbzz\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.104952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.104918 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd50076-b058-41af-8482-638f0fed4601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ed7ae5-hqbzz\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.105175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.105148 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd50076-b058-41af-8482-638f0fed4601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ed7ae5-hqbzz\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.205767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.205730 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k99f\" (UniqueName: \"kubernetes.io/projected/0dd50076-b058-41af-8482-638f0fed4601-kube-api-access-6k99f\") pod \"maas-api-revoke-sub-e0ed7ae5-hqbzz\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.205918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.205779 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd50076-b058-41af-8482-638f0fed4601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ed7ae5-hqbzz\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.205976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.205933 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd50076-b058-41af-8482-638f0fed4601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ed7ae5-hqbzz\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.206599 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.206581 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd50076-b058-41af-8482-638f0fed4601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ed7ae5-hqbzz\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.208162 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.208145 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd50076-b058-41af-8482-638f0fed4601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ed7ae5-hqbzz\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.213451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.213428 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k99f\" (UniqueName: \"kubernetes.io/projected/0dd50076-b058-41af-8482-638f0fed4601-kube-api-access-6k99f\") pod \"maas-api-revoke-sub-e0ed7ae5-hqbzz\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.313890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.313850 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:41.518834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.518797 6287 generic.go:356] "Generic (PLEG): container finished" podID="23a442ac-8dac-4589-be51-54cd2265aa5e" containerID="ddc97e5497aa7e373793386a2614fc785fba9b6e20563e8ed16a8ddafbbb68fd" exitCode=0 Nov 01 22:33:41.519222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.518847 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" event={"ID":"23a442ac-8dac-4589-be51-54cd2265aa5e","Type":"ContainerDied","Data":"ddc97e5497aa7e373793386a2614fc785fba9b6e20563e8ed16a8ddafbbb68fd"} Nov 01 22:33:41.520298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.520274 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" event={"ID":"fc8534fc-3426-4c34-bc85-dc1dc63e5d06","Type":"ContainerDied","Data":"6b037143c110a7c6f421cc7ec48d4fbcec32d1d10cf72df296f30b6fae83bf32"} Nov 01 22:33:41.520384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.520301 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8" Nov 01 22:33:41.520384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.520320 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b037143c110a7c6f421cc7ec48d4fbcec32d1d10cf72df296f30b6fae83bf32" Nov 01 22:33:41.601407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.596952 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz"] Nov 01 22:33:41.601407 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:41.600433 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd50076_b058_41af_8482_638f0fed4601.slice/crio-fd55db978a816e13cf22a22cdb03df1569c90c61bc5f1def906094bebc96254e WatchSource:0}: Error finding container fd55db978a816e13cf22a22cdb03df1569c90c61bc5f1def906094bebc96254e: Status 404 returned error can't find the container with id fd55db978a816e13cf22a22cdb03df1569c90c61bc5f1def906094bebc96254e Nov 01 22:33:41.666965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.666940 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:41.811829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.811791 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-xks-ca\") pod \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " Nov 01 22:33:41.811990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.811906 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-kube-api-access-9j6bq\" (UniqueName: \"kubernetes.io/projected/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-kube-api-access-9j6bq\") pod \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " Nov 01 22:33:41.811990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.811935 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-openshift-ca\") pod \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\" (UID: \"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4\") " Nov 01 22:33:41.812395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.812365 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-openshift-ca" pod "7b3db5fb-3ee8-4279-8888-14b8e6cc0df4" (UID: "7b3db5fb-3ee8-4279-8888-14b8e6cc0df4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:41.813841 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.813820 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-xks-ca" pod "7b3db5fb-3ee8-4279-8888-14b8e6cc0df4" (UID: "7b3db5fb-3ee8-4279-8888-14b8e6cc0df4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:41.813947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.813886 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-kube-api-access-9j6bq" pod "7b3db5fb-3ee8-4279-8888-14b8e6cc0df4" (UID: "7b3db5fb-3ee8-4279-8888-14b8e6cc0df4"). InnerVolumeSpecName "kube-api-access-9j6bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:41.913627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.913600 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j6bq\" (UniqueName: \"kubernetes.io/projected/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-kube-api-access-9j6bq\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:41.913627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.913627 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:41.913780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:41.913638 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:42.528254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.528227 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" Nov 01 22:33:42.528254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.528238 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt" event={"ID":"7b3db5fb-3ee8-4279-8888-14b8e6cc0df4","Type":"ContainerDied","Data":"8fac8da1806dfbf4dd85df1b71a1c7b8589e8acb34b7b6665edf8d2037d79787"} Nov 01 22:33:42.528723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.528280 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fac8da1806dfbf4dd85df1b71a1c7b8589e8acb34b7b6665edf8d2037d79787" Nov 01 22:33:42.530039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.530010 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" event={"ID":"0dd50076-b058-41af-8482-638f0fed4601","Type":"ContainerStarted","Data":"8fdb1fedd06125644e8961a8c0aaee078d833deb33f35159457f4efbeddedc80"} Nov 01 22:33:42.530180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.530057 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" event={"ID":"0dd50076-b058-41af-8482-638f0fed4601","Type":"ContainerStarted","Data":"fd55db978a816e13cf22a22cdb03df1569c90c61bc5f1def906094bebc96254e"} Nov 01 22:33:42.546812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.546773 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" podStartSLOduration=2.546759537 podStartE2EDuration="2.546759537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:42.544915309 +0000 UTC m=+635.193414554" watchObservedRunningTime="2026-11-01 22:33:42.546759537 +0000 UTC m=+635.195258789" Nov 01 22:33:42.682320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.682294 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:42.824025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.823942 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-xks-ca\") pod \"23a442ac-8dac-4589-be51-54cd2265aa5e\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " Nov 01 22:33:42.824025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.823979 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-openshift-ca\") pod \"23a442ac-8dac-4589-be51-54cd2265aa5e\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " Nov 01 22:33:42.824025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.824013 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a442ac-8dac-4589-be51-54cd2265aa5e-kube-api-access-s6wpc\" (UniqueName: \"kubernetes.io/projected/23a442ac-8dac-4589-be51-54cd2265aa5e-kube-api-access-s6wpc\") pod \"23a442ac-8dac-4589-be51-54cd2265aa5e\" (UID: \"23a442ac-8dac-4589-be51-54cd2265aa5e\") " Nov 01 22:33:42.824436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.824408 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-openshift-ca" pod "23a442ac-8dac-4589-be51-54cd2265aa5e" (UID: "23a442ac-8dac-4589-be51-54cd2265aa5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:42.826180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.826155 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-xks-ca" pod "23a442ac-8dac-4589-be51-54cd2265aa5e" (UID: "23a442ac-8dac-4589-be51-54cd2265aa5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:42.826354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.826330 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a442ac-8dac-4589-be51-54cd2265aa5e-kube-api-access-s6wpc" pod "23a442ac-8dac-4589-be51-54cd2265aa5e" (UID: "23a442ac-8dac-4589-be51-54cd2265aa5e"). InnerVolumeSpecName "kube-api-access-s6wpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:42.925017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.924982 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:42.925017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.925013 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a442ac-8dac-4589-be51-54cd2265aa5e-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:42.925017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:42.925026 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6wpc\" (UniqueName: \"kubernetes.io/projected/23a442ac-8dac-4589-be51-54cd2265aa5e-kube-api-access-s6wpc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:43.537867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:43.537837 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" Nov 01 22:33:43.538331 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:43.537867 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt" event={"ID":"23a442ac-8dac-4589-be51-54cd2265aa5e","Type":"ContainerDied","Data":"0053174bb6cdde05deff2625394f7911452eb9789108cee2ce2576ff56b2556f"} Nov 01 22:33:43.538331 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:43.537925 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0053174bb6cdde05deff2625394f7911452eb9789108cee2ce2576ff56b2556f" Nov 01 22:33:43.540747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:43.540724 6287 generic.go:356] "Generic (PLEG): container finished" podID="0dd50076-b058-41af-8482-638f0fed4601" containerID="8fdb1fedd06125644e8961a8c0aaee078d833deb33f35159457f4efbeddedc80" exitCode=0 Nov 01 22:33:43.540867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:43.540805 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" event={"ID":"0dd50076-b058-41af-8482-638f0fed4601","Type":"ContainerDied","Data":"8fdb1fedd06125644e8961a8c0aaee078d833deb33f35159457f4efbeddedc80"} Nov 01 22:33:44.691766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.691736 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:44.849495 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.849409 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd50076-b058-41af-8482-638f0fed4601-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd50076-b058-41af-8482-638f0fed4601-maas-api-xks-ca\") pod \"0dd50076-b058-41af-8482-638f0fed4601\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " Nov 01 22:33:44.849650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.849532 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd50076-b058-41af-8482-638f0fed4601-kube-api-access-6k99f\" (UniqueName: \"kubernetes.io/projected/0dd50076-b058-41af-8482-638f0fed4601-kube-api-access-6k99f\") pod \"0dd50076-b058-41af-8482-638f0fed4601\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " Nov 01 22:33:44.849650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.849579 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd50076-b058-41af-8482-638f0fed4601-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd50076-b058-41af-8482-638f0fed4601-maas-api-openshift-ca\") pod \"0dd50076-b058-41af-8482-638f0fed4601\" (UID: \"0dd50076-b058-41af-8482-638f0fed4601\") " Nov 01 22:33:44.850014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.849986 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd50076-b058-41af-8482-638f0fed4601-maas-api-openshift-ca" pod "0dd50076-b058-41af-8482-638f0fed4601" (UID: "0dd50076-b058-41af-8482-638f0fed4601"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:44.851832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.851803 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd50076-b058-41af-8482-638f0fed4601-maas-api-xks-ca" pod "0dd50076-b058-41af-8482-638f0fed4601" (UID: "0dd50076-b058-41af-8482-638f0fed4601"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:44.852025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.852006 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd50076-b058-41af-8482-638f0fed4601-kube-api-access-6k99f" pod "0dd50076-b058-41af-8482-638f0fed4601" (UID: "0dd50076-b058-41af-8482-638f0fed4601"). InnerVolumeSpecName "kube-api-access-6k99f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:44.951091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.951052 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k99f\" (UniqueName: \"kubernetes.io/projected/0dd50076-b058-41af-8482-638f0fed4601-kube-api-access-6k99f\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:44.951091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.951084 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd50076-b058-41af-8482-638f0fed4601-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:44.951091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:44.951096 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd50076-b058-41af-8482-638f0fed4601-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:45.557721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:45.557695 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" Nov 01 22:33:45.557912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:45.557738 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz" event={"ID":"0dd50076-b058-41af-8482-638f0fed4601","Type":"ContainerDied","Data":"fd55db978a816e13cf22a22cdb03df1569c90c61bc5f1def906094bebc96254e"} Nov 01 22:33:45.557912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:45.557787 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd55db978a816e13cf22a22cdb03df1569c90c61bc5f1def906094bebc96254e" Nov 01 22:33:45.596268 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:33:45.596232 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:33:47.406597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.406512 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm"] Nov 01 22:33:47.407310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.407289 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0dd50076-b058-41af-8482-638f0fed4601" containerName="revoke-keys" Nov 01 22:33:47.407413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.407311 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd50076-b058-41af-8482-638f0fed4601" containerName="revoke-keys" Nov 01 22:33:47.407413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.407339 6287 cpu_manager.go:414] "containerMap: removing container" podUID="23a442ac-8dac-4589-be51-54cd2265aa5e" containerName="revoke-keys" Nov 01 22:33:47.407413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.407349 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a442ac-8dac-4589-be51-54cd2265aa5e" containerName="revoke-keys" Nov 01 22:33:47.407413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.407380 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7b3db5fb-3ee8-4279-8888-14b8e6cc0df4" containerName="revoke-keys" Nov 01 22:33:47.407413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.407387 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3db5fb-3ee8-4279-8888-14b8e6cc0df4" containerName="revoke-keys" Nov 01 22:33:47.407660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.407514 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3db5fb-3ee8-4279-8888-14b8e6cc0df4" containerName="revoke-keys" Nov 01 22:33:47.407660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.407530 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd50076-b058-41af-8482-638f0fed4601" containerName="revoke-keys" Nov 01 22:33:47.407660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.407544 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a442ac-8dac-4589-be51-54cd2265aa5e" containerName="revoke-keys" Nov 01 22:33:47.411535 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.411510 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.422900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.421480 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:33:47.422900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.421984 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:33:47.422900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.422065 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:33:47.422900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.422132 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:33:47.431140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.427286 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm"] Nov 01 22:33:47.477559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.477527 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d67d9e1-57mxm\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.477747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.477604 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fnv\" (UniqueName: \"kubernetes.io/projected/ef16090b-a3af-4a95-bcb8-fcff2385c39a-kube-api-access-z2fnv\") pod \"maas-api-revoke-sub-5d67d9e1-57mxm\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.477747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.477659 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d67d9e1-57mxm\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.578559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.578519 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d67d9e1-57mxm\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.578757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.578662 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fnv\" (UniqueName: \"kubernetes.io/projected/ef16090b-a3af-4a95-bcb8-fcff2385c39a-kube-api-access-z2fnv\") pod \"maas-api-revoke-sub-5d67d9e1-57mxm\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.578757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.578710 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d67d9e1-57mxm\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.581288 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.579454 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d67d9e1-57mxm\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.582211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.582186 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d67d9e1-57mxm\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.599976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.599951 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fnv\" (UniqueName: \"kubernetes.io/projected/ef16090b-a3af-4a95-bcb8-fcff2385c39a-kube-api-access-z2fnv\") pod \"maas-api-revoke-sub-5d67d9e1-57mxm\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:47.746594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:47.746554 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:48.064999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:48.050378 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm"] Nov 01 22:33:48.069508 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:48.069473 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef16090b_a3af_4a95_bcb8_fcff2385c39a.slice/crio-e69b66218018fdd934887bd75df5dd408b5779fdd2442721ace83757d2088022 WatchSource:0}: Error finding container e69b66218018fdd934887bd75df5dd408b5779fdd2442721ace83757d2088022: Status 404 returned error can't find the container with id e69b66218018fdd934887bd75df5dd408b5779fdd2442721ace83757d2088022 Nov 01 22:33:48.599793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:48.598778 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" event={"ID":"ef16090b-a3af-4a95-bcb8-fcff2385c39a","Type":"ContainerStarted","Data":"6a8dc3d9f94e3e57a048436c31c073b24a57340f1d1876b6664c4d0bbc9a02b1"} Nov 01 22:33:48.599793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:48.598856 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" event={"ID":"ef16090b-a3af-4a95-bcb8-fcff2385c39a","Type":"ContainerStarted","Data":"e69b66218018fdd934887bd75df5dd408b5779fdd2442721ace83757d2088022"} Nov 01 22:33:48.676188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:48.642027 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" podStartSLOduration=1.6420047819999999 podStartE2EDuration="1.642004782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:48.61918514 +0000 UTC m=+641.267684375" watchObservedRunningTime="2026-11-01 22:33:48.642004782 +0000 UTC m=+641.290504018" Nov 01 22:33:49.616146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.612935 6287 generic.go:356] "Generic (PLEG): container finished" podID="ef16090b-a3af-4a95-bcb8-fcff2385c39a" containerID="6a8dc3d9f94e3e57a048436c31c073b24a57340f1d1876b6664c4d0bbc9a02b1" exitCode=0 Nov 01 22:33:49.616146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.613187 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" event={"ID":"ef16090b-a3af-4a95-bcb8-fcff2385c39a","Type":"ContainerDied","Data":"6a8dc3d9f94e3e57a048436c31c073b24a57340f1d1876b6664c4d0bbc9a02b1"} Nov 01 22:33:49.790957 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.790917 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x"] Nov 01 22:33:49.795224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.795202 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:49.807312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.807284 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x"] Nov 01 22:33:49.944675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.944591 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:49.954963 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.954940 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9782305-bg47x\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:49.955098 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.954966 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xsc\" (UniqueName: \"kubernetes.io/projected/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-kube-api-access-p6xsc\") pod \"maas-api-revoke-sub-b9782305-bg47x\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:49.955161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.955133 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9782305-bg47x\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:49.957597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:49.957578 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:50.056470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.056438 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9782305-bg47x\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:50.056675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.056551 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9782305-bg47x\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:50.056675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.056570 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xsc\" (UniqueName: \"kubernetes.io/projected/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-kube-api-access-p6xsc\") pod \"maas-api-revoke-sub-b9782305-bg47x\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:50.057270 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.057244 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9782305-bg47x\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:50.059233 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.059201 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9782305-bg47x\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:50.064996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.064967 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xsc\" (UniqueName: \"kubernetes.io/projected/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-kube-api-access-p6xsc\") pod \"maas-api-revoke-sub-b9782305-bg47x\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:50.116769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.116747 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:50.393700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.393668 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x"] Nov 01 22:33:50.394098 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:50.394071 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc345eedb_8a1b_43db_8680_9aa4e2cac4f2.slice/crio-2ede8b8b4d23f805bf870abf050a3db925947179e6bcf60b5cbdf932d5a8a010 WatchSource:0}: Error finding container 2ede8b8b4d23f805bf870abf050a3db925947179e6bcf60b5cbdf932d5a8a010: Status 404 returned error can't find the container with id 2ede8b8b4d23f805bf870abf050a3db925947179e6bcf60b5cbdf932d5a8a010 Nov 01 22:33:50.621788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.621758 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" event={"ID":"c345eedb-8a1b-43db-8680-9aa4e2cac4f2","Type":"ContainerStarted","Data":"2ede8b8b4d23f805bf870abf050a3db925947179e6bcf60b5cbdf932d5a8a010"} Nov 01 22:33:50.636569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.636541 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:33:50.775480 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.775456 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:50.966305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.966225 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-openshift-ca\") pod \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " Nov 01 22:33:50.966451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.966318 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-xks-ca\") pod \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " Nov 01 22:33:50.966451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.966391 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef16090b-a3af-4a95-bcb8-fcff2385c39a-kube-api-access-z2fnv\" (UniqueName: \"kubernetes.io/projected/ef16090b-a3af-4a95-bcb8-fcff2385c39a-kube-api-access-z2fnv\") pod \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\" (UID: \"ef16090b-a3af-4a95-bcb8-fcff2385c39a\") " Nov 01 22:33:50.966587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.966566 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-openshift-ca" pod "ef16090b-a3af-4a95-bcb8-fcff2385c39a" (UID: "ef16090b-a3af-4a95-bcb8-fcff2385c39a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:50.966797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.966783 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:50.968431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.968407 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-xks-ca" pod "ef16090b-a3af-4a95-bcb8-fcff2385c39a" (UID: "ef16090b-a3af-4a95-bcb8-fcff2385c39a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:50.968542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:50.968458 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef16090b-a3af-4a95-bcb8-fcff2385c39a-kube-api-access-z2fnv" pod "ef16090b-a3af-4a95-bcb8-fcff2385c39a" (UID: "ef16090b-a3af-4a95-bcb8-fcff2385c39a"). InnerVolumeSpecName "kube-api-access-z2fnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:51.067498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:51.067381 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2fnv\" (UniqueName: \"kubernetes.io/projected/ef16090b-a3af-4a95-bcb8-fcff2385c39a-kube-api-access-z2fnv\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:51.067498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:51.067406 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef16090b-a3af-4a95-bcb8-fcff2385c39a-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:51.630540 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:51.630510 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" Nov 01 22:33:51.631003 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:51.630548 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm" event={"ID":"ef16090b-a3af-4a95-bcb8-fcff2385c39a","Type":"ContainerDied","Data":"e69b66218018fdd934887bd75df5dd408b5779fdd2442721ace83757d2088022"} Nov 01 22:33:51.631003 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:51.630599 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e69b66218018fdd934887bd75df5dd408b5779fdd2442721ace83757d2088022" Nov 01 22:33:51.632664 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:51.632634 6287 generic.go:356] "Generic (PLEG): container finished" podID="c345eedb-8a1b-43db-8680-9aa4e2cac4f2" containerID="92d7865bb78c2387e24101e9465077c0fbeda191cb2b8936fdbc0ec15b3000f3" exitCode=0 Nov 01 22:33:51.632796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:51.632728 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" event={"ID":"c345eedb-8a1b-43db-8680-9aa4e2cac4f2","Type":"ContainerDied","Data":"92d7865bb78c2387e24101e9465077c0fbeda191cb2b8936fdbc0ec15b3000f3"} Nov 01 22:33:52.687836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.687796 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86"] Nov 01 22:33:52.688390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.688372 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ef16090b-a3af-4a95-bcb8-fcff2385c39a" containerName="revoke-keys" Nov 01 22:33:52.688448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.688391 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef16090b-a3af-4a95-bcb8-fcff2385c39a" containerName="revoke-keys" Nov 01 22:33:52.688497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.688487 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef16090b-a3af-4a95-bcb8-fcff2385c39a" containerName="revoke-keys" Nov 01 22:33:52.691960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.691938 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.718490 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.706933 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86"] Nov 01 22:33:52.783334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.783311 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:52.786532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.786512 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d3a9811-9jd86\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.786594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.786561 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d3a9811-9jd86\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.786754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.786736 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmgkp\" (UniqueName: \"kubernetes.io/projected/da65e844-1634-4d66-abce-b559ee60d4a2-kube-api-access-tmgkp\") pod \"maas-api-revoke-sub-2d3a9811-9jd86\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.838501 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.838471 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm"] Nov 01 22:33:52.838985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.838968 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c345eedb-8a1b-43db-8680-9aa4e2cac4f2" containerName="revoke-keys" Nov 01 22:33:52.838985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.838982 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c345eedb-8a1b-43db-8680-9aa4e2cac4f2" containerName="revoke-keys" Nov 01 22:33:52.839175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.839094 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c345eedb-8a1b-43db-8680-9aa4e2cac4f2" containerName="revoke-keys" Nov 01 22:33:52.842754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.842735 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.861211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.861185 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm"] Nov 01 22:33:52.887212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887180 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-kube-api-access-p6xsc\" (UniqueName: \"kubernetes.io/projected/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-kube-api-access-p6xsc\") pod \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " Nov 01 22:33:52.887358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887248 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-openshift-ca\") pod \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " Nov 01 22:33:52.887358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887324 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-xks-ca\") pod \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\" (UID: \"c345eedb-8a1b-43db-8680-9aa4e2cac4f2\") " Nov 01 22:33:52.887891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887654 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.887891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887711 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d3a9811-9jd86\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.887891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887738 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.887891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887763 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c9a796-525c-464f-8f26-0317ae2657d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.887891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887832 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tgh\" (UniqueName: \"kubernetes.io/projected/97c9a796-525c-464f-8f26-0317ae2657d7-kube-api-access-l7tgh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.887891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887870 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.888208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887897 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.888208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.887992 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c9a796-525c-464f-8f26-0317ae2657d7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.888208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.888037 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c9a796-525c-464f-8f26-0317ae2657d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.888208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.888081 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmgkp\" (UniqueName: \"kubernetes.io/projected/da65e844-1634-4d66-abce-b559ee60d4a2-kube-api-access-tmgkp\") pod \"maas-api-revoke-sub-2d3a9811-9jd86\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.888208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.888162 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.888461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.888233 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d3a9811-9jd86\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.889469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.889440 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d3a9811-9jd86\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.890140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.890096 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-openshift-ca" pod "c345eedb-8a1b-43db-8680-9aa4e2cac4f2" (UID: "c345eedb-8a1b-43db-8680-9aa4e2cac4f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:52.892151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.891183 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-xks-ca" pod "c345eedb-8a1b-43db-8680-9aa4e2cac4f2" (UID: "c345eedb-8a1b-43db-8680-9aa4e2cac4f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:52.892151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.891198 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d3a9811-9jd86\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.892441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.892414 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-kube-api-access-p6xsc" pod "c345eedb-8a1b-43db-8680-9aa4e2cac4f2" (UID: "c345eedb-8a1b-43db-8680-9aa4e2cac4f2"). InnerVolumeSpecName "kube-api-access-p6xsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:52.901990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.901967 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmgkp\" (UniqueName: \"kubernetes.io/projected/da65e844-1634-4d66-abce-b559ee60d4a2-kube-api-access-tmgkp\") pod \"maas-api-revoke-sub-2d3a9811-9jd86\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:52.989630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.989594 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c9a796-525c-464f-8f26-0317ae2657d7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.989789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.989647 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c9a796-525c-464f-8f26-0317ae2657d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.989789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.989698 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.989907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.989780 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.989907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.989839 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.989907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.989863 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c9a796-525c-464f-8f26-0317ae2657d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.990063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.989929 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tgh\" (UniqueName: \"kubernetes.io/projected/97c9a796-525c-464f-8f26-0317ae2657d7-kube-api-access-l7tgh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.990063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.989969 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.990063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.989997 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.990264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.990107 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:52.990264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.990162 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6xsc\" (UniqueName: \"kubernetes.io/projected/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-kube-api-access-p6xsc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:52.990264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.990178 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c345eedb-8a1b-43db-8680-9aa4e2cac4f2-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:52.990264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.990212 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.990264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.990251 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.990513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.990268 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.990513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.990396 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97c9a796-525c-464f-8f26-0317ae2657d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.990513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.990485 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.992367 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.992341 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97c9a796-525c-464f-8f26-0317ae2657d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.992471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.992385 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97c9a796-525c-464f-8f26-0317ae2657d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.997885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.997857 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97c9a796-525c-464f-8f26-0317ae2657d7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:52.997978 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:52.997941 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tgh\" (UniqueName: \"kubernetes.io/projected/97c9a796-525c-464f-8f26-0317ae2657d7-kube-api-access-l7tgh\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vt5hm\" (UID: \"97c9a796-525c-464f-8f26-0317ae2657d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:53.013814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.013795 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:53.165295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.165218 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:53.291680 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:53.291650 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda65e844_1634_4d66_abce_b559ee60d4a2.slice/crio-29915232c6c99afba863426accc8c623b4dfc9820b7294255b6bf035996f0144 WatchSource:0}: Error finding container 29915232c6c99afba863426accc8c623b4dfc9820b7294255b6bf035996f0144: Status 404 returned error can't find the container with id 29915232c6c99afba863426accc8c623b4dfc9820b7294255b6bf035996f0144 Nov 01 22:33:53.291812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.291706 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86"] Nov 01 22:33:53.444771 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:53.444736 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c9a796_525c_464f_8f26_0317ae2657d7.slice/crio-93798f70cbf8125dda7d7bc779c3e5acfe5212efff49c66de392f1e7710c0021 WatchSource:0}: Error finding container 93798f70cbf8125dda7d7bc779c3e5acfe5212efff49c66de392f1e7710c0021: Status 404 returned error can't find the container with id 93798f70cbf8125dda7d7bc779c3e5acfe5212efff49c66de392f1e7710c0021 Nov 01 22:33:53.444973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.444952 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm"] Nov 01 22:33:53.447855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.447821 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:33:53.447945 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.447886 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:33:53.447945 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.447914 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:33:53.652342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.652303 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" event={"ID":"da65e844-1634-4d66-abce-b559ee60d4a2","Type":"ContainerStarted","Data":"29915232c6c99afba863426accc8c623b4dfc9820b7294255b6bf035996f0144"} Nov 01 22:33:53.657035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.654168 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" event={"ID":"c345eedb-8a1b-43db-8680-9aa4e2cac4f2","Type":"ContainerDied","Data":"2ede8b8b4d23f805bf870abf050a3db925947179e6bcf60b5cbdf932d5a8a010"} Nov 01 22:33:53.657035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.654218 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ede8b8b4d23f805bf870abf050a3db925947179e6bcf60b5cbdf932d5a8a010" Nov 01 22:33:53.657035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.654176 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x" Nov 01 22:33:53.665155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.664071 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" event={"ID":"97c9a796-525c-464f-8f26-0317ae2657d7","Type":"ContainerStarted","Data":"ca5568291af9d35f9489fa91f5060cea52af7c8c8cd4a5cd7035e7bb35cff886"} Nov 01 22:33:53.665155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.664159 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" event={"ID":"97c9a796-525c-464f-8f26-0317ae2657d7","Type":"ContainerStarted","Data":"93798f70cbf8125dda7d7bc779c3e5acfe5212efff49c66de392f1e7710c0021"} Nov 01 22:33:53.687583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.687537 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podStartSLOduration=1.687523578 podStartE2EDuration="1.687523578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:53.682512437 +0000 UTC m=+646.331011673" watchObservedRunningTime="2026-11-01 22:33:53.687523578 +0000 UTC m=+646.336022812" Nov 01 22:33:53.847402 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.847365 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5"] Nov 01 22:33:53.852446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.852420 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:53.858802 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.857889 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-aa441b\"/\"distinct-2-w5-aa441b-kserve-self-signed-certs\"" Nov 01 22:33:53.859679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.859505 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5"] Nov 01 22:33:53.900219 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.900190 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbnk\" (UniqueName: \"kubernetes.io/projected/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-kube-api-access-nxbnk\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:53.900367 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.900347 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tls-certs\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:53.900421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.900387 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-home\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:53.900421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.900417 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-dshm\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:53.900497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.900442 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tmp-dir\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:53.900530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:53.900504 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-model-cache\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.001356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.001310 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tls-certs\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.001356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.001359 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-home\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.001617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.001393 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-dshm\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.001617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.001417 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tmp-dir\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.001617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.001439 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-model-cache\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.001617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.001532 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbnk\" (UniqueName: \"kubernetes.io/projected/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-kube-api-access-nxbnk\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.001936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.001898 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tmp-dir\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.002140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.002096 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-model-cache\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.002247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.002222 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-home\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.004573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.004535 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tls-certs\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.004725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.004627 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-dshm\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.011013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.010987 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbnk\" (UniqueName: \"kubernetes.io/projected/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-kube-api-access-nxbnk\") pod \"distinct-2-w5-aa441b-kserve-554cd9786-sdmc5\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.165603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.165504 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:33:54.166900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.166863 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 22:33:54.167174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.166917 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 22:33:54.173051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.173028 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:54.449455 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:54.449429 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05a6224f_c5a4_4ed1_8cbb_b1b4216a6f97.slice/crio-5baa8dc05dd964e78cce45a01c73d167cf328875efa7022b623e0e197401fe6b WatchSource:0}: Error finding container 5baa8dc05dd964e78cce45a01c73d167cf328875efa7022b623e0e197401fe6b: Status 404 returned error can't find the container with id 5baa8dc05dd964e78cce45a01c73d167cf328875efa7022b623e0e197401fe6b Nov 01 22:33:54.450713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.450690 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5"] Nov 01 22:33:54.697309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.697214 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" event={"ID":"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97","Type":"ContainerStarted","Data":"1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c"} Nov 01 22:33:54.697547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.697520 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" event={"ID":"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97","Type":"ContainerStarted","Data":"5baa8dc05dd964e78cce45a01c73d167cf328875efa7022b623e0e197401fe6b"} Nov 01 22:33:54.699108 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.699081 6287 generic.go:356] "Generic (PLEG): container finished" podID="da65e844-1634-4d66-abce-b559ee60d4a2" containerID="39855cfb735256b2770b2c2dc663d3ba4d0b9da25cba1f1d4daaa92adde53a6c" exitCode=0 Nov 01 22:33:54.699236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.699198 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" event={"ID":"da65e844-1634-4d66-abce-b559ee60d4a2","Type":"ContainerDied","Data":"39855cfb735256b2770b2c2dc663d3ba4d0b9da25cba1f1d4daaa92adde53a6c"} Nov 01 22:33:54.721477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:54.721431 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podStartSLOduration=1.721417614 podStartE2EDuration="1.721417614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:54.719003771 +0000 UTC m=+647.367503007" watchObservedRunningTime="2026-11-01 22:33:54.721417614 +0000 UTC m=+647.369916848" Nov 01 22:33:55.046653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.046618 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4"] Nov 01 22:33:55.050904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.050885 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.058403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.058380 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4"] Nov 01 22:33:55.123711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.123677 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab2777f2-bjgm4\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.123874 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.123794 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab2777f2-bjgm4\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.123938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.123908 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hn7\" (UniqueName: \"kubernetes.io/projected/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-kube-api-access-97hn7\") pod \"maas-api-revoke-sub-ab2777f2-bjgm4\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.165826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.165791 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 22:33:55.165992 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.165856 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 22:33:55.182147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.175270 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:33:55.225392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.225357 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab2777f2-bjgm4\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.225608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.225415 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab2777f2-bjgm4\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.225608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.225541 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hn7\" (UniqueName: \"kubernetes.io/projected/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-kube-api-access-97hn7\") pod \"maas-api-revoke-sub-ab2777f2-bjgm4\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.226215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.226192 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab2777f2-bjgm4\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.228006 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.227984 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab2777f2-bjgm4\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.235639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.235618 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hn7\" (UniqueName: \"kubernetes.io/projected/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-kube-api-access-97hn7\") pod \"maas-api-revoke-sub-ab2777f2-bjgm4\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.368695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.368617 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:55.695622 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:55.695581 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d4f8f6_8e81_4b83_87aa_fb8e280edc32.slice/crio-9bfe1fcece3e93d0673f5ff515086082f7927acb8051afd744e621dd768f3712 WatchSource:0}: Error finding container 9bfe1fcece3e93d0673f5ff515086082f7927acb8051afd744e621dd768f3712: Status 404 returned error can't find the container with id 9bfe1fcece3e93d0673f5ff515086082f7927acb8051afd744e621dd768f3712 Nov 01 22:33:55.696189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.696161 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4"] Nov 01 22:33:55.709205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.709172 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" event={"ID":"53d4f8f6-8e81-4b83-87aa-fb8e280edc32","Type":"ContainerStarted","Data":"9bfe1fcece3e93d0673f5ff515086082f7927acb8051afd744e621dd768f3712"} Nov 01 22:33:55.921905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.921260 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:55.936131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.936079 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da65e844-1634-4d66-abce-b559ee60d4a2-kube-api-access-tmgkp\" (UniqueName: \"kubernetes.io/projected/da65e844-1634-4d66-abce-b559ee60d4a2-kube-api-access-tmgkp\") pod \"da65e844-1634-4d66-abce-b559ee60d4a2\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " Nov 01 22:33:55.936253 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.936176 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-xks-ca\") pod \"da65e844-1634-4d66-abce-b559ee60d4a2\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " Nov 01 22:33:55.936304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.936280 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-openshift-ca\") pod \"da65e844-1634-4d66-abce-b559ee60d4a2\" (UID: \"da65e844-1634-4d66-abce-b559ee60d4a2\") " Nov 01 22:33:55.936634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.936612 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-openshift-ca" pod "da65e844-1634-4d66-abce-b559ee60d4a2" (UID: "da65e844-1634-4d66-abce-b559ee60d4a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:55.936778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.936762 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:55.938126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.938095 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-xks-ca" pod "da65e844-1634-4d66-abce-b559ee60d4a2" (UID: "da65e844-1634-4d66-abce-b559ee60d4a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:55.938216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:55.938159 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da65e844-1634-4d66-abce-b559ee60d4a2-kube-api-access-tmgkp" pod "da65e844-1634-4d66-abce-b559ee60d4a2" (UID: "da65e844-1634-4d66-abce-b559ee60d4a2"). InnerVolumeSpecName "kube-api-access-tmgkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:56.037842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:56.037812 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmgkp\" (UniqueName: \"kubernetes.io/projected/da65e844-1634-4d66-abce-b559ee60d4a2-kube-api-access-tmgkp\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:56.037842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:56.037837 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da65e844-1634-4d66-abce-b559ee60d4a2-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:56.165852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:56.165817 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 22:33:56.166256 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:56.165873 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 22:33:56.717973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:56.717951 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" Nov 01 22:33:56.718247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:56.717950 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86" event={"ID":"da65e844-1634-4d66-abce-b559ee60d4a2","Type":"ContainerDied","Data":"29915232c6c99afba863426accc8c623b4dfc9820b7294255b6bf035996f0144"} Nov 01 22:33:56.718356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:56.718279 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29915232c6c99afba863426accc8c623b4dfc9820b7294255b6bf035996f0144" Nov 01 22:33:56.719684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:56.719660 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" event={"ID":"53d4f8f6-8e81-4b83-87aa-fb8e280edc32","Type":"ContainerStarted","Data":"3263f907320b021f87886779cced718ea84ba716e3debb2ff697179afe3e8d7a"} Nov 01 22:33:56.738962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:56.738925 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" podStartSLOduration=1.738913416 podStartE2EDuration="1.738913416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:56.737240991 +0000 UTC m=+649.385740240" watchObservedRunningTime="2026-11-01 22:33:56.738913416 +0000 UTC m=+649.387412676" Nov 01 22:33:57.167387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:57.166545 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 22:33:57.167387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:57.166605 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 22:33:57.728544 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:57.728508 6287 generic.go:356] "Generic (PLEG): container finished" podID="53d4f8f6-8e81-4b83-87aa-fb8e280edc32" containerID="3263f907320b021f87886779cced718ea84ba716e3debb2ff697179afe3e8d7a" exitCode=0 Nov 01 22:33:57.728725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:57.728593 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" event={"ID":"53d4f8f6-8e81-4b83-87aa-fb8e280edc32","Type":"ContainerDied","Data":"3263f907320b021f87886779cced718ea84ba716e3debb2ff697179afe3e8d7a"} Nov 01 22:33:58.165667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:58.165631 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 22:33:58.165845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:58.165697 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 22:33:58.938940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:58.938606 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:58.998184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:58.974472 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-xks-ca\") pod \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " Nov 01 22:33:58.998184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:58.974532 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-openshift-ca\") pod \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " Nov 01 22:33:58.998184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:58.974588 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-kube-api-access-97hn7\" (UniqueName: \"kubernetes.io/projected/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-kube-api-access-97hn7\") pod \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\" (UID: \"53d4f8f6-8e81-4b83-87aa-fb8e280edc32\") " Nov 01 22:33:59.005226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.002941 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-openshift-ca" pod "53d4f8f6-8e81-4b83-87aa-fb8e280edc32" (UID: "53d4f8f6-8e81-4b83-87aa-fb8e280edc32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:33:59.008045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.008009 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-kube-api-access-97hn7" pod "53d4f8f6-8e81-4b83-87aa-fb8e280edc32" (UID: "53d4f8f6-8e81-4b83-87aa-fb8e280edc32"). InnerVolumeSpecName "kube-api-access-97hn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:59.009013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.008972 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-xks-ca" pod "53d4f8f6-8e81-4b83-87aa-fb8e280edc32" (UID: "53d4f8f6-8e81-4b83-87aa-fb8e280edc32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:59.026191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.026166 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls"] Nov 01 22:33:59.026689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.026673 6287 cpu_manager.go:414] "containerMap: removing container" podUID="53d4f8f6-8e81-4b83-87aa-fb8e280edc32" containerName="revoke-keys" Nov 01 22:33:59.026689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.026688 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d4f8f6-8e81-4b83-87aa-fb8e280edc32" containerName="revoke-keys" Nov 01 22:33:59.026833 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.026724 6287 cpu_manager.go:414] "containerMap: removing container" podUID="da65e844-1634-4d66-abce-b559ee60d4a2" containerName="revoke-keys" Nov 01 22:33:59.026833 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.026729 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da65e844-1634-4d66-abce-b559ee60d4a2" containerName="revoke-keys" Nov 01 22:33:59.026833 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.026824 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="da65e844-1634-4d66-abce-b559ee60d4a2" containerName="revoke-keys" Nov 01 22:33:59.026997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.026837 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d4f8f6-8e81-4b83-87aa-fb8e280edc32" containerName="revoke-keys" Nov 01 22:33:59.030474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.030455 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.047398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.047374 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls"] Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.079776 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.079826 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9r5\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-kube-api-access-2t9r5\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.079863 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.079919 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.079944 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.079968 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.080017 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.080069 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6f8907-e8c1-493e-9a47-6aaa669e3796-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.080192 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.080282 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.080299 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:59.085137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.080315 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97hn7\" (UniqueName: \"kubernetes.io/projected/53d4f8f6-8e81-4b83-87aa-fb8e280edc32-kube-api-access-97hn7\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:59.166056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.166017 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 22:33:59.166258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.166065 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 22:33:59.195891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.195797 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.195922 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.195951 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9r5\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-kube-api-access-2t9r5\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.196221 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.196429 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.196506 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196562 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.196540 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.196565 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196638 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.196620 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.196822 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.196879 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.196967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.196914 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.197033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.197014 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6f8907-e8c1-493e-9a47-6aaa669e3796-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.197636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.197614 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6f8907-e8c1-493e-9a47-6aaa669e3796-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.199412 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.199387 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.199605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.199587 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.211759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.211731 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.212252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.212235 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9r5\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-kube-api-access-2t9r5\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-7sgls\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.353943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.353859 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:33:59.556877 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:33:59.556844 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d4f8f6_8e81_4b83_87aa_fb8e280edc32.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:33:59.742463 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:33:59.742429 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6f8907_e8c1_493e_9a47_6aaa669e3796.slice/crio-afd6f351d44f8082a683cb94f785a25b67dd2e978f13cbf7ae021209d56cdf0a WatchSource:0}: Error finding container afd6f351d44f8082a683cb94f785a25b67dd2e978f13cbf7ae021209d56cdf0a: Status 404 returned error can't find the container with id afd6f351d44f8082a683cb94f785a25b67dd2e978f13cbf7ae021209d56cdf0a Nov 01 22:33:59.746427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.743893 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls"] Nov 01 22:33:59.759657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.759533 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" event={"ID":"53d4f8f6-8e81-4b83-87aa-fb8e280edc32","Type":"ContainerDied","Data":"9bfe1fcece3e93d0673f5ff515086082f7927acb8051afd744e621dd768f3712"} Nov 01 22:33:59.759755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.759667 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4" Nov 01 22:33:59.759755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.759711 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bfe1fcece3e93d0673f5ff515086082f7927acb8051afd744e621dd768f3712" Nov 01 22:33:59.759869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.759837 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:33:59.759936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.759923 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:33:59.759972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:33:59.759955 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:00.166227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:00.166129 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 22:34:00.166227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:00.166188 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 22:34:00.768027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:00.767990 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" event={"ID":"de6f8907-e8c1-493e-9a47-6aaa669e3796","Type":"ContainerStarted","Data":"2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184"} Nov 01 22:34:00.768300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:00.768064 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" event={"ID":"de6f8907-e8c1-493e-9a47-6aaa669e3796","Type":"ContainerStarted","Data":"afd6f351d44f8082a683cb94f785a25b67dd2e978f13cbf7ae021209d56cdf0a"} Nov 01 22:34:00.802744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:00.802668 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podStartSLOduration=2.802649124 podStartE2EDuration="2.802649124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:00.799989615 +0000 UTC m=+653.448488850" watchObservedRunningTime="2026-11-01 22:34:00.802649124 +0000 UTC m=+653.451148359" Nov 01 22:34:01.166028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:01.165945 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 22:34:01.166028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:01.166006 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 22:34:01.354182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:01.354145 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:34:01.355816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:01.355789 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-7sgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:34:01.356062 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:01.355843 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 01 22:34:02.354910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:02.354876 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-7sgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:34:02.355458 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:02.354929 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 01 22:34:02.543058 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:02.543022 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:34:02.543873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:02.543853 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:34:02.544718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:02.544700 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" Nov 01 22:34:03.355130 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.355092 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-7sgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:34:03.355513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.355161 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 01 22:34:03.828415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.828377 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv"] Nov 01 22:34:03.833297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.833269 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:03.836990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.836967 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3d8490\"/\"openshift-service-ca.crt\"" Nov 01 22:34:03.839778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.839755 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3d8490\"/\"kube-root-ca.crt\"" Nov 01 22:34:03.840889 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.840864 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3d8490\"/\"distinct-2-w2-3d8490-kserve-self-signed-certs\"" Nov 01 22:34:03.841020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.840927 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3d8490\"/\"default-dockercfg-8gvtb\"" Nov 01 22:34:03.841882 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.841862 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv"] Nov 01 22:34:03.967012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.966974 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-tmp-dir\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:03.967207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.967051 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-model-cache\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:03.967207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.967158 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-dshm\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:03.967286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.967199 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-home\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:03.967338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.967325 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2s9\" (UniqueName: \"kubernetes.io/projected/ca59259c-14aa-4db9-b627-d72c97bbdc61-kube-api-access-pv2s9\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:03.967382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:03.967368 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca59259c-14aa-4db9-b627-d72c97bbdc61-tls-certs\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.068460 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.068426 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-tmp-dir\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.068645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.068523 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-model-cache\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.068645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.068564 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-dshm\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.068645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.068597 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-home\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.068837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.068676 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2s9\" (UniqueName: \"kubernetes.io/projected/ca59259c-14aa-4db9-b627-d72c97bbdc61-kube-api-access-pv2s9\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.068837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.068704 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca59259c-14aa-4db9-b627-d72c97bbdc61-tls-certs\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.069034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.069007 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-tmp-dir\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.069431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.069408 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-model-cache\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.069639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.069613 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-home\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.085187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.076733 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca59259c-14aa-4db9-b627-d72c97bbdc61-tls-certs\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.110140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.090900 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2s9\" (UniqueName: \"kubernetes.io/projected/ca59259c-14aa-4db9-b627-d72c97bbdc61-kube-api-access-pv2s9\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.110140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.095568 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-dshm\") pod \"distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.147984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.147951 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:04.198134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.189299 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:34:04.222066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.221800 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:34:04.355421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.355336 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-7sgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:34:04.355421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.355401 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 01 22:34:04.500607 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:04.500577 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca59259c_14aa_4db9_b627_d72c97bbdc61.slice/crio-05a39dbfc930704f10979d2a6d20d24f20763bcc237a0eeaeccdf5cdd0f1e523 WatchSource:0}: Error finding container 05a39dbfc930704f10979d2a6d20d24f20763bcc237a0eeaeccdf5cdd0f1e523: Status 404 returned error can't find the container with id 05a39dbfc930704f10979d2a6d20d24f20763bcc237a0eeaeccdf5cdd0f1e523 Nov 01 22:34:04.501571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.501549 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv"] Nov 01 22:34:04.804979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.804945 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c"] Nov 01 22:34:04.809156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.809105 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:04.813033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.813009 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:34:04.813181 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.813077 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:34:04.814065 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.814045 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:34:04.814290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.814271 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:34:04.817665 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.817641 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c"] Nov 01 22:34:04.819371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.819346 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" event={"ID":"ca59259c-14aa-4db9-b627-d72c97bbdc61","Type":"ContainerStarted","Data":"addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b"} Nov 01 22:34:04.819468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.819389 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" event={"ID":"ca59259c-14aa-4db9-b627-d72c97bbdc61","Type":"ContainerStarted","Data":"05a39dbfc930704f10979d2a6d20d24f20763bcc237a0eeaeccdf5cdd0f1e523"} Nov 01 22:34:04.832925 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.832902 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:34:04.842479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.842429 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podStartSLOduration=1.842413293 podStartE2EDuration="1.842413293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:04.841712513 +0000 UTC m=+657.490211750" watchObservedRunningTime="2026-11-01 22:34:04.842413293 +0000 UTC m=+657.490912535" Nov 01 22:34:04.901144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.901008 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-111df7ad-l5j9c\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:04.901345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.901165 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-111df7ad-l5j9c\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:04.901345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:04.901272 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqmr\" (UniqueName: \"kubernetes.io/projected/874c0e5a-ade2-4aa8-8265-6a604777fc8d-kube-api-access-fsqmr\") pod \"maas-api-revoke-sub-111df7ad-l5j9c\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:05.003136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.002837 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-111df7ad-l5j9c\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:05.003136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.002898 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-111df7ad-l5j9c\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:05.003136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.002977 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqmr\" (UniqueName: \"kubernetes.io/projected/874c0e5a-ade2-4aa8-8265-6a604777fc8d-kube-api-access-fsqmr\") pod \"maas-api-revoke-sub-111df7ad-l5j9c\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:05.004210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.004182 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-111df7ad-l5j9c\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:05.006903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.006873 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-111df7ad-l5j9c\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:05.011957 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.011928 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqmr\" (UniqueName: \"kubernetes.io/projected/874c0e5a-ade2-4aa8-8265-6a604777fc8d-kube-api-access-fsqmr\") pod \"maas-api-revoke-sub-111df7ad-l5j9c\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:05.127642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.127474 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:05.148344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.148320 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:05.354959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.354920 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-7sgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:34:05.355144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.354979 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 01 22:34:05.432491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.432466 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c"] Nov 01 22:34:05.433030 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:05.433010 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874c0e5a_ade2_4aa8_8265_6a604777fc8d.slice/crio-969ba4bb354f8d9fdc11733f811334a87b0b3487afa988bbcfd74904bdde6dbd WatchSource:0}: Error finding container 969ba4bb354f8d9fdc11733f811334a87b0b3487afa988bbcfd74904bdde6dbd: Status 404 returned error can't find the container with id 969ba4bb354f8d9fdc11733f811334a87b0b3487afa988bbcfd74904bdde6dbd Nov 01 22:34:05.831351 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:05.830504 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" event={"ID":"874c0e5a-ade2-4aa8-8265-6a604777fc8d","Type":"ContainerStarted","Data":"969ba4bb354f8d9fdc11733f811334a87b0b3487afa988bbcfd74904bdde6dbd"} Nov 01 22:34:06.585672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:06.585628 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:34:06.586989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:06.586967 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:34:06.587946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:06.587926 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:34:06.840048 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:06.840010 6287 generic.go:356] "Generic (PLEG): container finished" podID="874c0e5a-ade2-4aa8-8265-6a604777fc8d" containerID="c3d99a07e5cf4c4f5b543f84db3efac5d7d73b7972dd944294e78a174567b39e" exitCode=0 Nov 01 22:34:06.840247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:06.840097 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" event={"ID":"874c0e5a-ade2-4aa8-8265-6a604777fc8d","Type":"ContainerDied","Data":"c3d99a07e5cf4c4f5b543f84db3efac5d7d73b7972dd944294e78a174567b39e"} Nov 01 22:34:07.973951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:07.973914 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp"] Nov 01 22:34:07.980756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:07.980732 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:07.985867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:07.985775 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e25182\"/\"distinct-2-w0-e25182-kserve-self-signed-certs\"" Nov 01 22:34:07.988629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:07.988312 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp"] Nov 01 22:34:08.002651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.002626 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:08.142695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.142607 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-xks-ca\") pod \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " Nov 01 22:34:08.143102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.143000 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874c0e5a-ade2-4aa8-8265-6a604777fc8d-kube-api-access-fsqmr\" (UniqueName: \"kubernetes.io/projected/874c0e5a-ade2-4aa8-8265-6a604777fc8d-kube-api-access-fsqmr\") pod \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " Nov 01 22:34:08.143102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.143093 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-openshift-ca\") pod \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\" (UID: \"874c0e5a-ade2-4aa8-8265-6a604777fc8d\") " Nov 01 22:34:08.143459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.143416 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-model-cache\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.143566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.143527 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb4781f-2260-4e03-8701-f60193ff81ce-tls-certs\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.143625 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.143604 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-tmp-dir\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.143745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.143723 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-dshm\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.143887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.143756 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8fd\" (UniqueName: \"kubernetes.io/projected/1fb4781f-2260-4e03-8701-f60193ff81ce-kube-api-access-2c8fd\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.143887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.143828 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-home\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.144011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.143887 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-openshift-ca" pod "874c0e5a-ade2-4aa8-8265-6a604777fc8d" (UID: "874c0e5a-ade2-4aa8-8265-6a604777fc8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:34:08.145392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.145364 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-xks-ca" pod "874c0e5a-ade2-4aa8-8265-6a604777fc8d" (UID: "874c0e5a-ade2-4aa8-8265-6a604777fc8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:34:08.145392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.145372 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874c0e5a-ade2-4aa8-8265-6a604777fc8d-kube-api-access-fsqmr" pod "874c0e5a-ade2-4aa8-8265-6a604777fc8d" (UID: "874c0e5a-ade2-4aa8-8265-6a604777fc8d"). InnerVolumeSpecName "kube-api-access-fsqmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:34:08.245475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245440 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-tmp-dir\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.245673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245536 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-dshm\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.245673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245564 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8fd\" (UniqueName: \"kubernetes.io/projected/1fb4781f-2260-4e03-8701-f60193ff81ce-kube-api-access-2c8fd\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.245673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245652 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-home\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.245864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245794 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-model-cache\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.245864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245833 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb4781f-2260-4e03-8701-f60193ff81ce-tls-certs\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.245966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245896 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-tmp-dir\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.245966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245926 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:08.245966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245942 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsqmr\" (UniqueName: \"kubernetes.io/projected/874c0e5a-ade2-4aa8-8265-6a604777fc8d-kube-api-access-fsqmr\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:08.245966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.245952 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874c0e5a-ade2-4aa8-8265-6a604777fc8d-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:08.246213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.246189 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-model-cache\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.246465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.246443 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-home\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.248236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.248208 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-dshm\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.248552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.248525 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb4781f-2260-4e03-8701-f60193ff81ce-tls-certs\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.253760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.253736 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8fd\" (UniqueName: \"kubernetes.io/projected/1fb4781f-2260-4e03-8701-f60193ff81ce-kube-api-access-2c8fd\") pod \"distinct-2-w0-e25182-kserve-75c6f87849-s7kfp\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.313537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.313505 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:08.594917 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:08.594889 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb4781f_2260_4e03_8701_f60193ff81ce.slice/crio-c27271805f168aa4ae8a4e72fb76bd23a4d42f209d76967cc345b76365e5aac2 WatchSource:0}: Error finding container c27271805f168aa4ae8a4e72fb76bd23a4d42f209d76967cc345b76365e5aac2: Status 404 returned error can't find the container with id c27271805f168aa4ae8a4e72fb76bd23a4d42f209d76967cc345b76365e5aac2 Nov 01 22:34:08.596914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.596883 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp"] Nov 01 22:34:08.854749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.854674 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" Nov 01 22:34:08.854749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.854703 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c" event={"ID":"874c0e5a-ade2-4aa8-8265-6a604777fc8d","Type":"ContainerDied","Data":"969ba4bb354f8d9fdc11733f811334a87b0b3487afa988bbcfd74904bdde6dbd"} Nov 01 22:34:08.854940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.854763 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="969ba4bb354f8d9fdc11733f811334a87b0b3487afa988bbcfd74904bdde6dbd" Nov 01 22:34:08.856429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.856395 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" event={"ID":"1fb4781f-2260-4e03-8701-f60193ff81ce","Type":"ContainerStarted","Data":"ffbd80ae6071549cf1ec7e3267a8718caa81f52fada557aa2f094355db606fd2"} Nov 01 22:34:08.856544 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.856436 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" event={"ID":"1fb4781f-2260-4e03-8701-f60193ff81ce","Type":"ContainerStarted","Data":"c27271805f168aa4ae8a4e72fb76bd23a4d42f209d76967cc345b76365e5aac2"} Nov 01 22:34:08.876296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:08.876254 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podStartSLOduration=1.876242303 podStartE2EDuration="1.876242303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:08.874951007 +0000 UTC m=+661.523450244" watchObservedRunningTime="2026-11-01 22:34:08.876242303 +0000 UTC m=+661.524741561" Nov 01 22:34:09.313588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:09.313561 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:10.245761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.245722 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p"] Nov 01 22:34:10.246621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.246594 6287 cpu_manager.go:414] "containerMap: removing container" podUID="874c0e5a-ade2-4aa8-8265-6a604777fc8d" containerName="revoke-keys" Nov 01 22:34:10.246621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.246622 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874c0e5a-ade2-4aa8-8265-6a604777fc8d" containerName="revoke-keys" Nov 01 22:34:10.246796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.246776 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="874c0e5a-ade2-4aa8-8265-6a604777fc8d" containerName="revoke-keys" Nov 01 22:34:10.251013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.250990 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.260405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.260384 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p"] Nov 01 22:34:10.263447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.263417 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b42cc4\"/\"distinct-w4-b42cc4-kserve-self-signed-certs\"" Nov 01 22:34:10.372574 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.372538 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-dshm\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.373020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.372610 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-tmp-dir\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.373020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.372657 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-home\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.373020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.372703 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e2c051-67dd-41c7-8301-e5b5697c868e-tls-certs\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.373020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.372745 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-model-cache\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.373020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.372961 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbcd\" (UniqueName: \"kubernetes.io/projected/29e2c051-67dd-41c7-8301-e5b5697c868e-kube-api-access-pdbcd\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.474476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.474430 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbcd\" (UniqueName: \"kubernetes.io/projected/29e2c051-67dd-41c7-8301-e5b5697c868e-kube-api-access-pdbcd\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.474673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.474484 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-dshm\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.474673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.474532 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-tmp-dir\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.474673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.474548 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-home\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.474673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.474604 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e2c051-67dd-41c7-8301-e5b5697c868e-tls-certs\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.474673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.474654 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-model-cache\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.475017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.474992 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-tmp-dir\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.475085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.475014 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-home\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.475162 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.475147 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-model-cache\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.477146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.477090 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-dshm\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.477464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.477443 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e2c051-67dd-41c7-8301-e5b5697c868e-tls-certs\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.485363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.485342 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbcd\" (UniqueName: \"kubernetes.io/projected/29e2c051-67dd-41c7-8301-e5b5697c868e-kube-api-access-pdbcd\") pod \"distinct-w4-b42cc4-kserve-558ccb4959-s6k2p\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.569753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.569676 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:10.868492 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:10.868452 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29e2c051_67dd_41c7_8301_e5b5697c868e.slice/crio-e8c826689799775ac8f12df1fbff9e78d31ffeb5364a94068b2e579eabca4227 WatchSource:0}: Error finding container e8c826689799775ac8f12df1fbff9e78d31ffeb5364a94068b2e579eabca4227: Status 404 returned error can't find the container with id e8c826689799775ac8f12df1fbff9e78d31ffeb5364a94068b2e579eabca4227 Nov 01 22:34:10.869091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.869065 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p"] Nov 01 22:34:10.882890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:10.882859 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" event={"ID":"29e2c051-67dd-41c7-8301-e5b5697c868e","Type":"ContainerStarted","Data":"e8c826689799775ac8f12df1fbff9e78d31ffeb5364a94068b2e579eabca4227"} Nov 01 22:34:11.899736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:11.899690 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" event={"ID":"29e2c051-67dd-41c7-8301-e5b5697c868e","Type":"ContainerStarted","Data":"9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70"} Nov 01 22:34:12.570717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:12.570681 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:13.964377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:13.964325 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podStartSLOduration=3.964310508 podStartE2EDuration="3.964310508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:11.924334032 +0000 UTC m=+664.572833269" watchObservedRunningTime="2026-11-01 22:34:13.964310508 +0000 UTC m=+666.612809772" Nov 01 22:34:13.967037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:13.967017 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f"] Nov 01 22:34:13.972507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:13.972490 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:13.984015 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:13.983988 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f"] Nov 01 22:34:14.141084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.141058 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.141084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.141086 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnnf\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-kube-api-access-4cnnf\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.141296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.141237 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.141296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.141275 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.141296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.141292 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.141442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.141416 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.141502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.141463 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.141564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.141547 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.141601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.141572 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a007c1-e42e-496b-b17a-fd657e79645d-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.149012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.148990 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:14.161308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.161290 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:14.242829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.242805 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.242968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.242865 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.242968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.242891 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.242968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.242926 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.242968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.242965 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.243229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.243037 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.243229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.243071 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a007c1-e42e-496b-b17a-fd657e79645d-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.243229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.243169 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.243229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.243195 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnnf\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-kube-api-access-4cnnf\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.244838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.244809 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a007c1-e42e-496b-b17a-fd657e79645d-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.245031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.245007 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.245190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.245125 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.245369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.245346 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.245487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.245463 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.246677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.246654 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.247617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.247596 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.250471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.250446 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr"] Nov 01 22:34:14.254738 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.254713 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnnf\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-kube-api-access-4cnnf\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.255466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.255444 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.255659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.255643 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.265026 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.265001 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr"] Nov 01 22:34:14.289806 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.289763 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:14.446331 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.446298 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.446331 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.446333 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05e16a-6810-4247-8326-231181aa3987-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.446532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.446439 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.446532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.446466 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6b2d\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-kube-api-access-q6b2d\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.446532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.446488 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.446644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.446589 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.446677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.446646 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.446715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.446697 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05e16a-6810-4247-8326-231181aa3987-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.446750 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.446724 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547430 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547585 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547618 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05e16a-6810-4247-8326-231181aa3987-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547679 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547705 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6b2d\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-kube-api-access-q6b2d\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547750 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547792 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547994 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547950 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547994 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547980 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.547994 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.547991 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.548107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.548058 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05e16a-6810-4247-8326-231181aa3987-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.548449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.548422 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.548577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.548506 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.548727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.548703 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05e16a-6810-4247-8326-231181aa3987-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.550204 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.550182 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.550441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.550422 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05e16a-6810-4247-8326-231181aa3987-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.555641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.555620 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.555722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.555677 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6b2d\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-kube-api-access-q6b2d\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.579075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.579057 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f"] Nov 01 22:34:14.580372 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:14.580348 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a007c1_e42e_496b_b17a_fd657e79645d.slice/crio-22d421edcdc50417c27d8a6b5486a8f1e5c6aa4ddab1db1fac4503a6667e37e3 WatchSource:0}: Error finding container 22d421edcdc50417c27d8a6b5486a8f1e5c6aa4ddab1db1fac4503a6667e37e3: Status 404 returned error can't find the container with id 22d421edcdc50417c27d8a6b5486a8f1e5c6aa4ddab1db1fac4503a6667e37e3 Nov 01 22:34:14.582753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.582720 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:14.582845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.582780 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:14.582845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.582811 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:14.597954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.597934 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:14.888640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.888616 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr"] Nov 01 22:34:14.889462 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:14.889421 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa05e16a_6810_4247_8326_231181aa3987.slice/crio-a369740fbf234b02ddc61552894ece6dae52db5d7321d6dedaa09143db29e8f5 WatchSource:0}: Error finding container a369740fbf234b02ddc61552894ece6dae52db5d7321d6dedaa09143db29e8f5: Status 404 returned error can't find the container with id a369740fbf234b02ddc61552894ece6dae52db5d7321d6dedaa09143db29e8f5 Nov 01 22:34:14.892373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.892319 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:14.892464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.892417 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:14.892464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.892453 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:14.923572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.923541 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" event={"ID":"a9a007c1-e42e-496b-b17a-fd657e79645d","Type":"ContainerStarted","Data":"eaa2d2e65024d43527fcbd061f9d567d6dc903304ff41fc3378e7295c6427385"} Nov 01 22:34:14.923658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.923610 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" event={"ID":"a9a007c1-e42e-496b-b17a-fd657e79645d","Type":"ContainerStarted","Data":"22d421edcdc50417c27d8a6b5486a8f1e5c6aa4ddab1db1fac4503a6667e37e3"} Nov 01 22:34:14.925166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.925138 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" event={"ID":"fa05e16a-6810-4247-8326-231181aa3987","Type":"ContainerStarted","Data":"a369740fbf234b02ddc61552894ece6dae52db5d7321d6dedaa09143db29e8f5"} Nov 01 22:34:14.948169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.948088 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podStartSLOduration=1.94807115 podStartE2EDuration="1.94807115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:14.945054477 +0000 UTC m=+667.593553713" watchObservedRunningTime="2026-11-01 22:34:14.94807115 +0000 UTC m=+667.596570386" Nov 01 22:34:14.949047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:14.949026 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:15.293002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:15.292970 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:15.295129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:15.295079 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:34:15.295315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:15.295290 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 22:34:15.617266 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:34:15.617177 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:34:15.934470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:15.934375 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" event={"ID":"fa05e16a-6810-4247-8326-231181aa3987","Type":"ContainerStarted","Data":"546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075"} Nov 01 22:34:15.958143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:15.958070 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podStartSLOduration=1.958055672 podStartE2EDuration="1.958055672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:15.955188485 +0000 UTC m=+668.603687721" watchObservedRunningTime="2026-11-01 22:34:15.958055672 +0000 UTC m=+668.606554907" Nov 01 22:34:16.295137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:16.294554 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:34:16.295137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:16.294634 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 22:34:16.599193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:16.598404 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:16.606136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:16.601300 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:34:16.606136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:16.601365 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 22:34:17.303137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:17.294785 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:34:17.303137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:17.294870 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 22:34:17.601085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:17.600545 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:34:17.601085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:17.600609 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 22:34:18.296138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:18.295250 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:34:18.296138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:18.295316 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 22:34:18.319137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:18.318628 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:18.333306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:18.333280 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:18.604352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:18.604261 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:34:18.604767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:18.604732 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 22:34:19.028995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:19.028962 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:19.293645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:19.293552 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:34:19.293915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:19.293884 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 22:34:19.608197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:19.605260 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:34:19.608197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:19.605319 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 22:34:20.291640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:20.291579 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:34:20.291968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:20.291936 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 22:34:20.588258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:20.588170 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:20.601462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:20.601426 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:34:20.601744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:20.601715 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 22:34:20.646043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:20.646012 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:21.023469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:21.023432 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:21.291405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:21.291310 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:34:21.291405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:21.291389 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 22:34:21.598996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:21.598906 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:34:21.598996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:21.598959 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 01 22:34:22.296082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:22.296048 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:22.296551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:22.296485 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:22.297378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:22.297358 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:34:22.603130 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:22.603035 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:22.603471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:22.603450 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:22.604001 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:22.603983 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:34:23.216551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.216519 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk"] Nov 01 22:34:23.220734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.220719 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.225328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.224957 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:34:23.225521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.225092 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:34:23.225521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.225244 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:34:23.226280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.226259 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:34:23.229304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.229279 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk"] Nov 01 22:34:23.302233 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.302201 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e701836c-gjfzk\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.302703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.302428 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e701836c-gjfzk\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.302703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.302469 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwfm\" (UniqueName: \"kubernetes.io/projected/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-kube-api-access-7kwfm\") pod \"maas-api-revoke-sub-e701836c-gjfzk\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.403461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.403427 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e701836c-gjfzk\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.403461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.403466 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwfm\" (UniqueName: \"kubernetes.io/projected/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-kube-api-access-7kwfm\") pod \"maas-api-revoke-sub-e701836c-gjfzk\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.403732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.403527 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e701836c-gjfzk\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.404174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.404151 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e701836c-gjfzk\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.405919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.405898 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e701836c-gjfzk\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.411678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.411656 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwfm\" (UniqueName: \"kubernetes.io/projected/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-kube-api-access-7kwfm\") pod \"maas-api-revoke-sub-e701836c-gjfzk\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.542453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.542426 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:23.836072 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:23.836043 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e84b0a5_504f_49ad_be1a_efbbe8ad8a2c.slice/crio-a5d427b918e431b7e486bb970d9a05cf5915bc0957cd21776a324a817ebfd10c WatchSource:0}: Error finding container a5d427b918e431b7e486bb970d9a05cf5915bc0957cd21776a324a817ebfd10c: Status 404 returned error can't find the container with id a5d427b918e431b7e486bb970d9a05cf5915bc0957cd21776a324a817ebfd10c Nov 01 22:34:23.836273 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:23.836253 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk"] Nov 01 22:34:24.039894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:24.039855 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" event={"ID":"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c","Type":"ContainerStarted","Data":"a5d427b918e431b7e486bb970d9a05cf5915bc0957cd21776a324a817ebfd10c"} Nov 01 22:34:25.050710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:25.050673 6287 generic.go:356] "Generic (PLEG): container finished" podID="7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c" containerID="509b2a227f00439912ece7466a836e134fc61e1e9585e4d9e05e29ee40ba1208" exitCode=0 Nov 01 22:34:25.051140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:25.050731 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" event={"ID":"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c","Type":"ContainerDied","Data":"509b2a227f00439912ece7466a836e134fc61e1e9585e4d9e05e29ee40ba1208"} Nov 01 22:34:26.219965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.219940 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:26.350593 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.350513 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-openshift-ca\") pod \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " Nov 01 22:34:26.350738 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.350658 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-kube-api-access-7kwfm\" (UniqueName: \"kubernetes.io/projected/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-kube-api-access-7kwfm\") pod \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " Nov 01 22:34:26.350738 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.350685 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-xks-ca\") pod \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\" (UID: \"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c\") " Nov 01 22:34:26.351024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.350992 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-openshift-ca" pod "7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c" (UID: "7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:34:26.351360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.351338 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:26.352851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.352828 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-xks-ca" pod "7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c" (UID: "7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:34:26.352924 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.352832 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-kube-api-access-7kwfm" pod "7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c" (UID: "7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c"). InnerVolumeSpecName "kube-api-access-7kwfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:34:26.453588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.453557 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwfm\" (UniqueName: \"kubernetes.io/projected/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-kube-api-access-7kwfm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:26.453588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:26.453584 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:27.066344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.066314 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" Nov 01 22:34:27.066551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.066352 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk" event={"ID":"7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c","Type":"ContainerDied","Data":"a5d427b918e431b7e486bb970d9a05cf5915bc0957cd21776a324a817ebfd10c"} Nov 01 22:34:27.066551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.066400 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d427b918e431b7e486bb970d9a05cf5915bc0957cd21776a324a817ebfd10c" Nov 01 22:34:27.546489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.546445 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl"] Nov 01 22:34:27.547368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.547342 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c" containerName="revoke-keys" Nov 01 22:34:27.547368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.547367 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c" containerName="revoke-keys" Nov 01 22:34:27.547843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.547516 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c" containerName="revoke-keys" Nov 01 22:34:27.551509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.551489 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.565526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.565499 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b28a40\"/\"openshift-service-ca.crt\"" Nov 01 22:34:27.565754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.565732 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b28a40\"/\"kube-root-ca.crt\"" Nov 01 22:34:27.566468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.566448 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b28a40\"/\"default-dockercfg-6lslv\"" Nov 01 22:34:27.566468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.566456 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b28a40\"/\"distinct-w6-b28a40-kserve-self-signed-certs\"" Nov 01 22:34:27.586834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.586806 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl"] Nov 01 22:34:27.671192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.671087 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-model-cache\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.671358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.671197 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-tmp-dir\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.671358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.671342 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b2092d-b96e-485d-922d-3322cafbcf4d-tls-certs\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.671447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.671376 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-home\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.671514 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.671492 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-dshm\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.671574 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.671536 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5z6h\" (UniqueName: \"kubernetes.io/projected/14b2092d-b96e-485d-922d-3322cafbcf4d-kube-api-access-w5z6h\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.773314 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.773272 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-model-cache\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.773488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.773333 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-tmp-dir\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.773576 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.773534 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b2092d-b96e-485d-922d-3322cafbcf4d-tls-certs\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.773576 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.773560 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-home\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.773690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.773672 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-dshm\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.773754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.773695 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5z6h\" (UniqueName: \"kubernetes.io/projected/14b2092d-b96e-485d-922d-3322cafbcf4d-kube-api-access-w5z6h\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.773804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.773752 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-model-cache\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.774420 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.774393 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-home\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.774565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.774512 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-tmp-dir\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.776604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.776576 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b2092d-b96e-485d-922d-3322cafbcf4d-tls-certs\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.776724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.776692 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-dshm\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.783277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.783250 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5z6h\" (UniqueName: \"kubernetes.io/projected/14b2092d-b96e-485d-922d-3322cafbcf4d-kube-api-access-w5z6h\") pod \"distinct-w6-b28a40-kserve-866497bdf6-xmtsl\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:27.867680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:27.867651 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:28.151290 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:28.151250 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14b2092d_b96e_485d_922d_3322cafbcf4d.slice/crio-69addb43f7be5325449aba35344136bc7142ad5df778df2aa207a742a0b6eb48 WatchSource:0}: Error finding container 69addb43f7be5325449aba35344136bc7142ad5df778df2aa207a742a0b6eb48: Status 404 returned error can't find the container with id 69addb43f7be5325449aba35344136bc7142ad5df778df2aa207a742a0b6eb48 Nov 01 22:34:28.151603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:28.151579 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl"] Nov 01 22:34:29.092983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:29.091883 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" event={"ID":"14b2092d-b96e-485d-922d-3322cafbcf4d","Type":"ContainerStarted","Data":"23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e"} Nov 01 22:34:29.092983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:29.091938 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" event={"ID":"14b2092d-b96e-485d-922d-3322cafbcf4d","Type":"ContainerStarted","Data":"69addb43f7be5325449aba35344136bc7142ad5df778df2aa207a742a0b6eb48"} Nov 01 22:34:29.137242 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:29.132104 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podStartSLOduration=2.13208376 podStartE2EDuration="2.13208376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:29.126407177 +0000 UTC m=+681.774906415" watchObservedRunningTime="2026-11-01 22:34:29.13208376 +0000 UTC m=+681.780582997" Nov 01 22:34:29.876714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:29.876644 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:30.833273 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:30.833229 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:30.936135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:30.935075 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:30.944324 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:30.944274 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Nov 01 22:34:30.975254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:30.975200 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.078138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.073193 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:34:31.248523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.248205 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.321127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.321067 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 01 22:34:31.366240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.366191 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.419260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.419207 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.449228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.449180 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:34:31.554317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.554208 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.574429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.574386 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded" Nov 01 22:34:31.594238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.594194 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.609210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.609172 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.643238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.643193 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.703238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.703194 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.718281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.718229 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:31.843676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.843559 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:34:31.844924 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.844897 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:34:32.002139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.977486 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:32.021138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.002997 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:32.164258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:31.996022 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:34:32.165871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.165844 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:32.226783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.226743 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:32.392080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.263102 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:32.393005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.369130 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:32.393614 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.393580 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:34:32.444198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.443288 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:34:32.444673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.444652 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:34:32.460467 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.460429 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:32.503348 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.471526 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:34:32.562149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.559667 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:32.562149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.560051 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:34:32.632164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.600955 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:32.632164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.603025 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:34:32.632164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.611733 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:32.632164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.612013 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:34:32.656541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.656516 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:34:32.661286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.657484 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:32.663828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.663435 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:34:32.664241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.664214 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:34:32.709243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.676035 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:34:32.711598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.710839 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:34:32.711598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.710914 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:32.711598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.710948 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:34:32.713745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.712191 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:32.714813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.714484 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:34:32.718829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.718802 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:34:32.720569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.720549 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:34:32.741145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.722909 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:32.741145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.722957 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:34:32.755612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.751807 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:34:32.760763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.760708 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:34:32.774986 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.774959 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j"] Nov 01 22:34:32.779458 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.779436 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j"] Nov 01 22:34:32.779594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.779536 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:32.784690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.784649 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 22:34:32.784848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.784829 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 22:34:32.784920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.784903 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 22:34:32.784979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.784919 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c7qwl\"" Nov 01 22:34:32.919819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.919782 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2957d06d-fec2-40f0-a4b4-b433ad6ac848-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:32.920349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.919909 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:32.920349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.919961 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:32.920510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.920342 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fbs\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-kube-api-access-22fbs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:32.920510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.920398 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:32.920510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.920484 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:32.920676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:32.920519 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.022214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.022171 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.022411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.022242 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fbs\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-kube-api-access-22fbs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.022411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.022295 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.022411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.022374 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.022411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.022404 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.022684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.022583 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2957d06d-fec2-40f0-a4b4-b433ad6ac848-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.022684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.022647 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.023953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.023722 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.029148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.025987 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2957d06d-fec2-40f0-a4b4-b433ad6ac848-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.029148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.026402 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.029148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.026482 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.029148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.026507 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.031379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.031354 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.036530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.036502 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fbs\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-kube-api-access-22fbs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bjq2j\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.097031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.096995 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:33.398146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.398104 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j"] Nov 01 22:34:33.398272 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:33.398105 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2957d06d_fec2_40f0_a4b4_b433ad6ac848.slice/crio-b1d234c5ee6dcbf0f2a47a4774bbab7c19334355283b3a045f86e25084c2cf80 WatchSource:0}: Error finding container b1d234c5ee6dcbf0f2a47a4774bbab7c19334355283b3a045f86e25084c2cf80: Status 404 returned error can't find the container with id b1d234c5ee6dcbf0f2a47a4774bbab7c19334355283b3a045f86e25084c2cf80 Nov 01 22:34:33.400512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.400494 6287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:34:33.724184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:33.722146 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" event={"ID":"2957d06d-fec2-40f0-a4b4-b433ad6ac848","Type":"ContainerStarted","Data":"b1d234c5ee6dcbf0f2a47a4774bbab7c19334355283b3a045f86e25084c2cf80"} Nov 01 22:34:36.681007 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:36.680971 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:36.681466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:36.681055 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:37.768364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.768323 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" event={"ID":"2957d06d-fec2-40f0-a4b4-b433ad6ac848","Type":"ContainerStarted","Data":"ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681"} Nov 01 22:34:37.768778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.768512 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:37.770377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.770353 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:34:37.790464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.790411 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" podStartSLOduration=2.5111712859999997 podStartE2EDuration="5.790397543s" totalImagesPullingTime="3.279226257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:37.78746942 +0000 UTC m=+690.435968679" watchObservedRunningTime="2026-11-01 22:34:37.790397543 +0000 UTC m=+690.438896778" Nov 01 22:34:37.863289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.863258 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn"] Nov 01 22:34:37.867534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.867517 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:37.867855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.867830 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:37.874796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.874773 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:34:37.874968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.874890 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:34:37.874968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.874944 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:34:37.875556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.875536 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:34:37.877960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.877936 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn"] Nov 01 22:34:37.885616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.885587 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a1729b4-f6xbn\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:37.885709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.885659 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a1729b4-f6xbn\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:37.885922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.885896 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66285\" (UniqueName: \"kubernetes.io/projected/3241b4f0-0d7b-4454-99d6-29697f4b84f6-kube-api-access-66285\") pod \"maas-api-revoke-sub-4a1729b4-f6xbn\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:37.886889 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.886855 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:37.986677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.986642 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a1729b4-f6xbn\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:37.986855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.986692 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a1729b4-f6xbn\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:37.986855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.986817 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66285\" (UniqueName: \"kubernetes.io/projected/3241b4f0-0d7b-4454-99d6-29697f4b84f6-kube-api-access-66285\") pod \"maas-api-revoke-sub-4a1729b4-f6xbn\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:37.987335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.987304 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a1729b4-f6xbn\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:37.989103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.989079 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a1729b4-f6xbn\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:37.996021 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:37.995998 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66285\" (UniqueName: \"kubernetes.io/projected/3241b4f0-0d7b-4454-99d6-29697f4b84f6-kube-api-access-66285\") pod \"maas-api-revoke-sub-4a1729b4-f6xbn\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:38.188267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:38.188183 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:38.475695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:38.475595 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn"] Nov 01 22:34:38.479345 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:38.479314 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3241b4f0_0d7b_4454_99d6_29697f4b84f6.slice/crio-99754dfa1ba55034e6ff77ee0408aa09f5973b209b62c1f8f61da8ad47a7a2e1 WatchSource:0}: Error finding container 99754dfa1ba55034e6ff77ee0408aa09f5973b209b62c1f8f61da8ad47a7a2e1: Status 404 returned error can't find the container with id 99754dfa1ba55034e6ff77ee0408aa09f5973b209b62c1f8f61da8ad47a7a2e1 Nov 01 22:34:38.778056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:38.778023 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" event={"ID":"3241b4f0-0d7b-4454-99d6-29697f4b84f6","Type":"ContainerStarted","Data":"99754dfa1ba55034e6ff77ee0408aa09f5973b209b62c1f8f61da8ad47a7a2e1"} Nov 01 22:34:38.789985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:38.789961 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:39.787019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:39.786981 6287 generic.go:356] "Generic (PLEG): container finished" podID="3241b4f0-0d7b-4454-99d6-29697f4b84f6" containerID="8d74f12abafa022ea1bf7baf0ebabc0a6dbceff960a5f60ddbe5dcfbd3b8ff8b" exitCode=0 Nov 01 22:34:39.787477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:39.787064 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" event={"ID":"3241b4f0-0d7b-4454-99d6-29697f4b84f6","Type":"ContainerDied","Data":"8d74f12abafa022ea1bf7baf0ebabc0a6dbceff960a5f60ddbe5dcfbd3b8ff8b"} Nov 01 22:34:40.934887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:40.934863 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:41.024446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.024414 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-xks-ca\") pod \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " Nov 01 22:34:41.024597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.024519 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3241b4f0-0d7b-4454-99d6-29697f4b84f6-kube-api-access-66285\" (UniqueName: \"kubernetes.io/projected/3241b4f0-0d7b-4454-99d6-29697f4b84f6-kube-api-access-66285\") pod \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " Nov 01 22:34:41.024710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.024698 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-openshift-ca\") pod \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\" (UID: \"3241b4f0-0d7b-4454-99d6-29697f4b84f6\") " Nov 01 22:34:41.025002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.024977 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-openshift-ca" pod "3241b4f0-0d7b-4454-99d6-29697f4b84f6" (UID: "3241b4f0-0d7b-4454-99d6-29697f4b84f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:34:41.026519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.026482 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-xks-ca" pod "3241b4f0-0d7b-4454-99d6-29697f4b84f6" (UID: "3241b4f0-0d7b-4454-99d6-29697f4b84f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:34:41.026629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.026539 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3241b4f0-0d7b-4454-99d6-29697f4b84f6-kube-api-access-66285" pod "3241b4f0-0d7b-4454-99d6-29697f4b84f6" (UID: "3241b4f0-0d7b-4454-99d6-29697f4b84f6"). InnerVolumeSpecName "kube-api-access-66285". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:34:41.126707 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.126638 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66285\" (UniqueName: \"kubernetes.io/projected/3241b4f0-0d7b-4454-99d6-29697f4b84f6-kube-api-access-66285\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:41.126707 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.126667 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:41.126707 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.126676 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3241b4f0-0d7b-4454-99d6-29697f4b84f6-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:41.804618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.804587 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" Nov 01 22:34:41.804618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.804597 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn" event={"ID":"3241b4f0-0d7b-4454-99d6-29697f4b84f6","Type":"ContainerDied","Data":"99754dfa1ba55034e6ff77ee0408aa09f5973b209b62c1f8f61da8ad47a7a2e1"} Nov 01 22:34:41.804835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:41.804672 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99754dfa1ba55034e6ff77ee0408aa09f5973b209b62c1f8f61da8ad47a7a2e1" Nov 01 22:34:44.091597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.091561 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226"] Nov 01 22:34:44.092400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.092373 6287 cpu_manager.go:414] "containerMap: removing container" podUID="3241b4f0-0d7b-4454-99d6-29697f4b84f6" containerName="revoke-keys" Nov 01 22:34:44.092400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.092400 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3241b4f0-0d7b-4454-99d6-29697f4b84f6" containerName="revoke-keys" Nov 01 22:34:44.092596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.092581 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3241b4f0-0d7b-4454-99d6-29697f4b84f6" containerName="revoke-keys" Nov 01 22:34:44.097434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.097412 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.117252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.117227 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226"] Nov 01 22:34:44.164443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.164415 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.164592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.164469 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.164592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.164538 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqr4k\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-kube-api-access-kqr4k\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.164671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.164589 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.164671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.164618 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.164746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.164690 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6383f0b-7182-4aae-80f8-0bb64da62924-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.164746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.164708 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.164809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.164786 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.164846 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.164809 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.265375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.265339 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw"] Nov 01 22:34:44.271817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.271783 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.272623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.272340 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.277596 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-data\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.278009 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-podinfo\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.272935 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqr4k\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-kube-api-access-kqr4k\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.278811 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.278856 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.278902 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6383f0b-7182-4aae-80f8-0bb64da62924-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.278925 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.278979 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.279002 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.280175 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-credential-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.280497 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-certs\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.281532 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-socket\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.284329 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6383f0b-7182-4aae-80f8-0bb64da62924-istiod-ca-cert\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.286620 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-envoy\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.290893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.290379 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqr4k\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-kube-api-access-kqr4k\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.298049 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.298027 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-token\") pod \"e2e-worker-w1-381659-openshift-default-894db6755-j5226\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.299366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.299349 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw"] Nov 01 22:34:44.299447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.299436 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.381375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.381300 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.381375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.381361 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad51f46-a415-4fad-9071-8ece456756e9-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.381580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.381438 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.381580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.381532 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.381744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.381668 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.381880 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.381745 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.381945 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.381869 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.381945 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.381914 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad51f46-a415-4fad-9071-8ece456756e9-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.382101 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.382065 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbmmp\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-kube-api-access-pbmmp\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.394557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.394533 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7"] Nov 01 22:34:44.398940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.398921 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.411301 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.411275 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7"] Nov 01 22:34:44.430137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.417950 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:44.470058 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.470030 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79"] Nov 01 22:34:44.475512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.475491 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483195 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483288 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483335 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483407 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483433 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483479 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483512 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483536 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483575 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad51f46-a415-4fad-9071-8ece456756e9-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483607 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbmmp\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-kube-api-access-pbmmp\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483675 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjsx8\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-kube-api-access-zjsx8\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.483751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483739 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.484827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483765 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.484827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483824 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad51f46-a415-4fad-9071-8ece456756e9-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.484827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483851 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.484827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483881 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.484827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.483931 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.484827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.484103 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79"] Nov 01 22:34:44.484827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.484579 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.484827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.484603 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.484827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.484703 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.485314 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.484995 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.485314 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.485069 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.485869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.485790 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad51f46-a415-4fad-9071-8ece456756e9-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.487653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.487626 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad51f46-a415-4fad-9071-8ece456756e9-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.491623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.488599 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.500600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.500571 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.500792 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.500764 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbmmp\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-kube-api-access-pbmmp\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.586489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586449 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.586672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586503 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.586672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586522 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.586672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586552 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.586672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586604 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjsx8\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-kube-api-access-zjsx8\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.586920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586704 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64d6k\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-kube-api-access-64d6k\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.586920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586878 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.587022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586927 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.587022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586882 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.587022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.586947 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.587149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587031 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.587149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587086 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.587149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587087 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.587149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587096 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.587316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587172 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.587316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587211 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.587316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587243 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac136e0f-6211-438b-8c16-67d56e89b4e3-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.587316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587280 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.587539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587333 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.587539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587432 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.587539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587459 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.587539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.587514 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.588049 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.588026 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.589454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.589426 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.589652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.589637 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.594865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.594834 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjsx8\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-kube-api-access-zjsx8\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.595237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.595221 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.634609 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.634541 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:44.689393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.688516 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.689393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.689066 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.689393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.689171 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.690660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.690635 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac136e0f-6211-438b-8c16-67d56e89b4e3-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.690782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.690739 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac136e0f-6211-438b-8c16-67d56e89b4e3-istiod-ca-cert\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.690826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.690806 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.690902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.690888 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.690946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.690914 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.690984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.690942 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.691023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.690996 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.691143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.691097 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64d6k\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-kube-api-access-64d6k\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.691199 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.691176 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-certs\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.691734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.691684 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-data\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.691890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.691868 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-credential-socket\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.693514 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.693350 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-envoy\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.694064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.694044 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-podinfo\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.701186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.701162 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64d6k\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-kube-api-access-64d6k\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.706012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.705985 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-token\") pod \"e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.726552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.724603 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:44.740340 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.732222 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226"] Nov 01 22:34:44.763423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.763386 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:44.763538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.763475 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:44.763671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.763651 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:44.811059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.810548 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:44.858468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.858395 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" event={"ID":"f6383f0b-7182-4aae-80f8-0bb64da62924","Type":"ContainerStarted","Data":"6d0fc536a4740c0e2417049d384bb8320aa7279c3cfc6f17442cd3ab0bb4c57e"} Nov 01 22:34:44.979255 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:44.979209 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad51f46_a415_4fad_9071_8ece456756e9.slice/crio-5f1cfe9cfc94085d40034e45c2e54d531d7fe33dae25db8a10795264f04acb3d WatchSource:0}: Error finding container 5f1cfe9cfc94085d40034e45c2e54d531d7fe33dae25db8a10795264f04acb3d: Status 404 returned error can't find the container with id 5f1cfe9cfc94085d40034e45c2e54d531d7fe33dae25db8a10795264f04acb3d Nov 01 22:34:44.983988 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.983201 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:44.983988 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.983285 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:44.983988 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.983333 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:44.995344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:44.995322 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw"] Nov 01 22:34:45.069189 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:45.069158 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee9df93_aaca_4b75_bc8b_f41d8519c256.slice/crio-c63c941962e6f1bc72ba737f3c23845237dc006f86d53c9b4b8ffaddf55d743d WatchSource:0}: Error finding container c63c941962e6f1bc72ba737f3c23845237dc006f86d53c9b4b8ffaddf55d743d: Status 404 returned error can't find the container with id c63c941962e6f1bc72ba737f3c23845237dc006f86d53c9b4b8ffaddf55d743d Nov 01 22:34:45.071587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.071558 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7"] Nov 01 22:34:45.071816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.071781 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:45.071911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.071870 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:45.071998 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.071946 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:45.142294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.142241 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:45.142880 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.142831 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:45.143056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.143034 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:45.161732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.161602 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79"] Nov 01 22:34:45.594503 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:34:45.594455 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:34:45.869674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.869583 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" event={"ID":"bad51f46-a415-4fad-9071-8ece456756e9","Type":"ContainerStarted","Data":"646b3135bf874c4ad6b772157a35e920daa0c47a7fa16c94c322fbd72615e44f"} Nov 01 22:34:45.869674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.869642 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" event={"ID":"bad51f46-a415-4fad-9071-8ece456756e9","Type":"ContainerStarted","Data":"5f1cfe9cfc94085d40034e45c2e54d531d7fe33dae25db8a10795264f04acb3d"} Nov 01 22:34:45.897917 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.897874 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" event={"ID":"f6383f0b-7182-4aae-80f8-0bb64da62924","Type":"ContainerStarted","Data":"a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490"} Nov 01 22:34:45.900593 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.900557 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" event={"ID":"ac136e0f-6211-438b-8c16-67d56e89b4e3","Type":"ContainerStarted","Data":"97a1253ce9a00c777281b5e6c318241730abc6a98f397460bcc1e8d828cef513"} Nov 01 22:34:45.900749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.900618 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" event={"ID":"ac136e0f-6211-438b-8c16-67d56e89b4e3","Type":"ContainerStarted","Data":"52474406ee1f227e53358035f1efe4f5804bb54efaa77ed58f9e824f29eedf76"} Nov 01 22:34:45.902651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.902621 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" event={"ID":"9ee9df93-aaca-4b75-bc8b-f41d8519c256","Type":"ContainerStarted","Data":"d1be23a457786c0a4719196b08e2054b7f729e2a3e3e4b2bd9a99a62452f18ff"} Nov 01 22:34:45.902770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.902677 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" event={"ID":"9ee9df93-aaca-4b75-bc8b-f41d8519c256","Type":"ContainerStarted","Data":"c63c941962e6f1bc72ba737f3c23845237dc006f86d53c9b4b8ffaddf55d743d"} Nov 01 22:34:45.904231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.904186 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podStartSLOduration=1.90417315 podStartE2EDuration="1.90417315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:45.898426747 +0000 UTC m=+698.546925983" watchObservedRunningTime="2026-11-01 22:34:45.90417315 +0000 UTC m=+698.552672386" Nov 01 22:34:45.918363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.918323 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podStartSLOduration=1.918310381 podStartE2EDuration="1.918310381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:45.916056165 +0000 UTC m=+698.564555394" watchObservedRunningTime="2026-11-01 22:34:45.918310381 +0000 UTC m=+698.566809615" Nov 01 22:34:45.937637 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.937590 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podStartSLOduration=1.937577168 podStartE2EDuration="1.937577168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:45.934207286 +0000 UTC m=+698.582706523" watchObservedRunningTime="2026-11-01 22:34:45.937577168 +0000 UTC m=+698.586076402" Nov 01 22:34:45.954657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:45.954611 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podStartSLOduration=1.954598141 podStartE2EDuration="1.954598141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:45.95106299 +0000 UTC m=+698.599562249" watchObservedRunningTime="2026-11-01 22:34:45.954598141 +0000 UTC m=+698.603097375" Nov 01 22:34:46.427138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.426411 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:46.427138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.426545 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-j5226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:34:46.427138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.426589 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 22:34:46.647297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.640698 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:46.649137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.648648 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 22:34:46.649137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.648712 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 22:34:46.727658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.725969 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:46.732141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.729203 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 22:34:46.732141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.729261 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 22:34:46.826999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.824375 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:46.847136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.835252 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:34:46.847136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:46.835317 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 22:34:47.422641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.422605 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-j5226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:34:47.422981 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.422938 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 22:34:47.645149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.642418 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 22:34:47.645149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.642482 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 22:34:47.662136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.661325 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:47.668234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.668196 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:47.717320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.717214 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:47.725323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.725288 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:34:47.726209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.726186 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 22:34:47.730199 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.730171 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 22:34:47.730352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.726498 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:34:47.734132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.726464 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:34:47.738133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.727325 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:47.764138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.727349 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:34:47.764710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.727385 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:34:47.765100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.727367 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:34:47.765452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.743373 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:47.765804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.759908 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:47.773133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.770257 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:47.794660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.794572 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:47.823136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.814439 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:34:47.823136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:47.814495 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 22:34:48.240249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.240199 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:48.422471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.422429 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-j5226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.439292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.439230 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 22:34:48.553384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.553280 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:48.683788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.683731 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:48.695256 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.695206 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:34:48.744140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.703618 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:34:48.769280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.769242 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:34:48.810191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.730200 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:34:48.819141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.733095 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.819412 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.819376 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 22:34:48.823134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.747219 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:48.828171 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.747186 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:48.829131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.793234 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:34:48.829393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.793257 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:48.829753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.793275 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:34:48.830161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.793293 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:34:48.830579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.792208 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:48.830995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.732804 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:48.831446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.795168 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:34:48.831853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.793194 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:48.832249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.733384 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 22:34:48.832428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.832402 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 22:34:48.832565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.828733 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:34:48.832746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.832732 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:34:48.850134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.832884 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:48.901190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.901148 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:34:48.901615 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.901598 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:34:48.901800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.901787 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:34:49.667765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.901498 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": context deadline exceeded" Nov 01 22:34:49.682307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:48.924408 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.682688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.682529 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 22:34:49.682853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.341571 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.707834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.466468 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-j5226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.708442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.708384 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 22:34:49.708709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.707704 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.716476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.701038 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.716729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.716378 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:34:49.743156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.730217 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.743156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.725271 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.743156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.741512 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 22:34:49.743156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.726901 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.743156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.741575 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 22:34:49.830136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.829480 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.830136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.829639 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.856031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.830894 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.860353 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.857208 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.860353 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.858358 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.986213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.864193 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:34:49.986213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.872261 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:34:49.986213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.872302 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.986213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.872373 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:34:49.986213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.924198 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 22:34:49.986213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.872821 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:34:49.986213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.872793 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:49.986213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:49.932769 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:34:50.069137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.043513 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:34:50.094595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.094567 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:50.098226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.098200 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:50.098471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.098457 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:34:50.566079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.566045 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-j5226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.566567 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.566429 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 22:34:50.658017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.657982 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.658345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.658294 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 22:34:50.730379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.730325 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:50.732403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.732370 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.732618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.732592 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 22:34:50.733881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.733850 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:34:50.734311 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.734286 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:34:50.734743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.734723 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:50.821229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.818709 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:34:50.821229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.818866 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 22:34:50.862408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.862348 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:50.863143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.863078 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:50.871379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.863348 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:50.900356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.900217 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:50.901151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.863395 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:50.945341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.945280 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:50.945848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.945818 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:34:50.963174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.945775 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:34:50.999135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:50.995311 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:51.039360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.039316 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:51.107839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.086875 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Nov 01 22:34:51.110797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.110762 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:34:51.406196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.358348 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:34:51.406196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.367904 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:51.711193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.710656 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded" Nov 01 22:34:51.758138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.756147 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.758138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.756215 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 22:34:51.758138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.757178 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.758138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.757216 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 22:34:51.758138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.757309 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:51.770770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.758853 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:34:51.770770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.759183 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:34:51.836138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.810838 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-j5226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.836138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.810918 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 22:34:51.881770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.881737 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:34:51.882033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.882001 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 22:34:51.882760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.882735 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:51.895135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.885420 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:52.143198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:51.889959 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:34:52.143198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.109199 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:52.143198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.118228 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:52.143198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.119440 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:34:52.236137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.150726 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:34:52.236137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.168816 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:52.236137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.209720 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:34:52.329372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.284877 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:34:52.329372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.285357 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:52.450190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.440566 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-j5226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.450190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.440633 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 22:34:52.564130 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:34:52.563144 6287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.084s" Nov 01 22:34:52.564130 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.563344 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:34:52.686837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.686255 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.686837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.686317 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 22:34:52.765571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.761258 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:34:52.793491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.793446 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.793860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.793746 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 22:34:52.794593 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.794536 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:52.802251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.802216 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:34:52.826138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.809595 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:34:52.868424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.867883 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:34:52.868424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.867934 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 22:34:52.886137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:52.883711 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:53.025193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.024385 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd"] Nov 01 22:34:53.070160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.058173 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.085995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.085699 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:34:53.085995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.085819 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:34:53.085995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.085834 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:34:53.085995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.085945 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:34:53.086760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.086740 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd"] Nov 01 22:34:53.087010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.086993 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:34:53.102573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.102216 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:34:53.123023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.120783 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:34:53.123023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.121515 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:34:53.123375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.123346 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:53.123761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.123735 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:34:53.160998 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.157567 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:53.323592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.189435 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:53.323592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.194534 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:34:53.323592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.204422 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:34:53.323592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.206786 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db502368-1c36-415a-b880-0f87ad31f871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc56f0bf-5bjxd\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.323592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.206823 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db502368-1c36-415a-b880-0f87ad31f871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc56f0bf-5bjxd\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.323592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.207129 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhfgb\" (UniqueName: \"kubernetes.io/projected/db502368-1c36-415a-b880-0f87ad31f871-kube-api-access-dhfgb\") pod \"maas-api-revoke-sub-dc56f0bf-5bjxd\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.323592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.232266 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:34:53.323592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.232729 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:34:53.326194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.325904 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhfgb\" (UniqueName: \"kubernetes.io/projected/db502368-1c36-415a-b880-0f87ad31f871-kube-api-access-dhfgb\") pod \"maas-api-revoke-sub-dc56f0bf-5bjxd\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.326194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.326015 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db502368-1c36-415a-b880-0f87ad31f871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc56f0bf-5bjxd\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.326194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.326042 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db502368-1c36-415a-b880-0f87ad31f871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc56f0bf-5bjxd\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.363752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.363572 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:34:53.364785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.364762 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:34:53.370024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.370004 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:34:53.374136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.370934 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db502368-1c36-415a-b880-0f87ad31f871-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc56f0bf-5bjxd\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.382618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.379360 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:34:53.419133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.418384 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-j5226 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 22:34:53.419133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.418435 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 22:34:53.450158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.434501 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:34:53.456402 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.451696 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhfgb\" (UniqueName: \"kubernetes.io/projected/db502368-1c36-415a-b880-0f87ad31f871-kube-api-access-dhfgb\") pod \"maas-api-revoke-sub-dc56f0bf-5bjxd\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.473660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.473588 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db502368-1c36-415a-b880-0f87ad31f871-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc56f0bf-5bjxd\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.499520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.498609 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:34:53.743652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.743369 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:53.843137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.820327 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:34:54.323202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:54.319411 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 22:34:54.323202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:53.832451 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 22:34:54.323202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:54.319518 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 22:34:54.444347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:54.443987 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:54.446919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:54.446272 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:54.448809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:54.447282 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:34:54.597189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:54.593317 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:54.744513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:54.728205 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:54.744513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:54.728280 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:55.119176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.095628 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:55.161343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.121063 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:55.177469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.177426 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:34:55.696189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.685235 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:55.749154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.748804 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:55.867260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.858241 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:34:55.867260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.858308 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:55.867260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.858615 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:55.901326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.868988 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:34:55.901326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.887652 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:55.901326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.888816 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:55.901326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.892067 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd"] Nov 01 22:34:55.901326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.895968 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:34:55.901326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.897484 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:55.901326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.898344 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:34:55.909392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.909348 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" event={"ID":"db502368-1c36-415a-b880-0f87ad31f871","Type":"ContainerStarted","Data":"eee8f245d3cee385499907cca8c54ce28f08ccab9721eb66e6510fa26b512825"} Nov 01 22:34:55.954000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:55.953471 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:34:56.942023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:56.941928 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" event={"ID":"db502368-1c36-415a-b880-0f87ad31f871","Type":"ContainerStarted","Data":"ee84588b4e57ec6eb4f9a1a625278ffafea70aff07cff30e273d3206086ee4c6"} Nov 01 22:34:56.976492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:56.976427 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" podStartSLOduration=5.976409369 podStartE2EDuration="5.976409369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:56.96946438 +0000 UTC m=+709.617963615" watchObservedRunningTime="2026-11-01 22:34:56.976409369 +0000 UTC m=+709.624908605" Nov 01 22:34:57.951931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:57.951896 6287 generic.go:356] "Generic (PLEG): container finished" podID="db502368-1c36-415a-b880-0f87ad31f871" containerID="ee84588b4e57ec6eb4f9a1a625278ffafea70aff07cff30e273d3206086ee4c6" exitCode=0 Nov 01 22:34:57.952377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:57.951981 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" event={"ID":"db502368-1c36-415a-b880-0f87ad31f871","Type":"ContainerDied","Data":"ee84588b4e57ec6eb4f9a1a625278ffafea70aff07cff30e273d3206086ee4c6"} Nov 01 22:34:59.134751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.134722 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:59.147767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.147739 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5"] Nov 01 22:34:59.148388 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.148371 6287 cpu_manager.go:414] "containerMap: removing container" podUID="db502368-1c36-415a-b880-0f87ad31f871" containerName="revoke-keys" Nov 01 22:34:59.148455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.148389 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db502368-1c36-415a-b880-0f87ad31f871" containerName="revoke-keys" Nov 01 22:34:59.148501 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.148492 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="db502368-1c36-415a-b880-0f87ad31f871" containerName="revoke-keys" Nov 01 22:34:59.152024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.152005 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.193572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.185374 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5"] Nov 01 22:34:59.308376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.308345 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db502368-1c36-415a-b880-0f87ad31f871-kube-api-access-dhfgb\" (UniqueName: \"kubernetes.io/projected/db502368-1c36-415a-b880-0f87ad31f871-kube-api-access-dhfgb\") pod \"db502368-1c36-415a-b880-0f87ad31f871\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " Nov 01 22:34:59.308571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.308443 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db502368-1c36-415a-b880-0f87ad31f871-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db502368-1c36-415a-b880-0f87ad31f871-maas-api-xks-ca\") pod \"db502368-1c36-415a-b880-0f87ad31f871\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " Nov 01 22:34:59.308571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.308531 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db502368-1c36-415a-b880-0f87ad31f871-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db502368-1c36-415a-b880-0f87ad31f871-maas-api-openshift-ca\") pod \"db502368-1c36-415a-b880-0f87ad31f871\" (UID: \"db502368-1c36-415a-b880-0f87ad31f871\") " Nov 01 22:34:59.308893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.308863 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.309019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.308911 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.309082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.309035 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db502368-1c36-415a-b880-0f87ad31f871-maas-api-openshift-ca" pod "db502368-1c36-415a-b880-0f87ad31f871" (UID: "db502368-1c36-415a-b880-0f87ad31f871"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:34:59.309082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.309064 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.309201 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.309166 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.309315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.309286 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.309440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.309329 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.309440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.309390 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.309440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.309421 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.309569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.309517 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5b9\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-kube-api-access-kk5b9\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.309718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.309704 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db502368-1c36-415a-b880-0f87ad31f871-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:59.310736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.310714 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db502368-1c36-415a-b880-0f87ad31f871-maas-api-xks-ca" pod "db502368-1c36-415a-b880-0f87ad31f871" (UID: "db502368-1c36-415a-b880-0f87ad31f871"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:34:59.310904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.310887 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db502368-1c36-415a-b880-0f87ad31f871-kube-api-access-dhfgb" pod "db502368-1c36-415a-b880-0f87ad31f871" (UID: "db502368-1c36-415a-b880-0f87ad31f871"). InnerVolumeSpecName "kube-api-access-dhfgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:34:59.332926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.332902 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk"] Nov 01 22:34:59.337358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.337341 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.364999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.364954 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk"] Nov 01 22:34:59.410946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.410913 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.410955 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411090 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411135 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411196 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411458 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411422 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-credential-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411461 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-certs\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411482 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-data\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411502 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411547 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411571 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411615 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5b9\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-kube-api-access-kk5b9\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.411853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411744 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhfgb\" (UniqueName: \"kubernetes.io/projected/db502368-1c36-415a-b880-0f87ad31f871-kube-api-access-dhfgb\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:59.411853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411761 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db502368-1c36-415a-b880-0f87ad31f871-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:59.411853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411814 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-socket\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.412035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.411975 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istiod-ca-cert\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.413895 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.413863 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-envoy\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.414080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.414063 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-podinfo\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.421262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.421236 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5b9\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-kube-api-access-kk5b9\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.422460 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.422439 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-token\") pod \"e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.466322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.466284 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:34:59.513703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.513662 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.513871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.513826 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.516989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.514169 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.516989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.514226 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.516989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.514265 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.516989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.514354 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.516989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.514427 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7zj\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-kube-api-access-wd7zj\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.516989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.514511 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.516989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.514550 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a234993f-449e-4d56-82c5-8bc2282a67fd-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.616376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.616286 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.616376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.616342 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a234993f-449e-4d56-82c5-8bc2282a67fd-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.616560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.616529 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.616596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.616564 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.616596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.616583 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.616713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.616603 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.616713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.616639 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.616814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.616741 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.616814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.616801 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7zj\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-kube-api-access-wd7zj\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.618430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.618342 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-credential-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.619867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.619382 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-socket\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.619867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.619424 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-certs\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.619867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.619535 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-data\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.619867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.619825 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a234993f-449e-4d56-82c5-8bc2282a67fd-istiod-ca-cert\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.620322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.620295 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-envoy\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.621646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.621626 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-podinfo\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.627416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.627397 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-token\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.630820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.630794 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7zj\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-kube-api-access-wd7zj\") pod \"e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.655091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.655068 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:34:59.773282 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:34:59.773228 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9100bc64_fc11_4947_aaf9_5c762fd53ac7.slice/crio-87e76022f2f2caca53f68a53b310c794613a603841bc11f6f092969fa7016642 WatchSource:0}: Error finding container 87e76022f2f2caca53f68a53b310c794613a603841bc11f6f092969fa7016642: Status 404 returned error can't find the container with id 87e76022f2f2caca53f68a53b310c794613a603841bc11f6f092969fa7016642 Nov 01 22:34:59.773962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.773934 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5"] Nov 01 22:34:59.776137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.776090 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:59.776223 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.776169 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:59.776223 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.776209 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:59.954020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.953993 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk"] Nov 01 22:34:59.957272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.957234 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:59.957390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.957324 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:59.957390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.957369 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:34:59.968976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.968949 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" event={"ID":"a234993f-449e-4d56-82c5-8bc2282a67fd","Type":"ContainerStarted","Data":"9dcf7c068283768473cfa5a75f166f0ecece68164811bd74dbda5df7e1416020"} Nov 01 22:34:59.970644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.970621 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" event={"ID":"db502368-1c36-415a-b880-0f87ad31f871","Type":"ContainerDied","Data":"eee8f245d3cee385499907cca8c54ce28f08ccab9721eb66e6510fa26b512825"} Nov 01 22:34:59.970755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.970667 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eee8f245d3cee385499907cca8c54ce28f08ccab9721eb66e6510fa26b512825" Nov 01 22:34:59.970755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.970631 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd" Nov 01 22:34:59.972462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.972427 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" event={"ID":"9100bc64-fc11-4947-aaf9-5c762fd53ac7","Type":"ContainerStarted","Data":"1c80a811a55c201d453f3f5b4f349cf9c31ff47adb110838d79102e0879a8c95"} Nov 01 22:34:59.972538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:34:59.972480 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" event={"ID":"9100bc64-fc11-4947-aaf9-5c762fd53ac7","Type":"ContainerStarted","Data":"87e76022f2f2caca53f68a53b310c794613a603841bc11f6f092969fa7016642"} Nov 01 22:35:00.005749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.005678 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podStartSLOduration=1.005622385 podStartE2EDuration="1.005622385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:00.003681498 +0000 UTC m=+712.652180737" watchObservedRunningTime="2026-11-01 22:35:00.005622385 +0000 UTC m=+712.654121625" Nov 01 22:35:00.060497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.060467 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9"] Nov 01 22:35:00.065652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.065623 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.077179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.073203 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:35:00.077179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.073264 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:00.077179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.073452 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:35:00.077179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.073747 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:35:00.089249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.089226 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9"] Nov 01 22:35:00.272215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.272179 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62292fe6-zlfk9\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.272797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.272773 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62292fe6-zlfk9\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.273164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.273146 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7q5z\" (UniqueName: \"kubernetes.io/projected/8504a501-0a9d-4127-b42e-e228b2767edd-kube-api-access-d7q5z\") pod \"maas-api-revoke-sub-62292fe6-zlfk9\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.381519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.381469 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7q5z\" (UniqueName: \"kubernetes.io/projected/8504a501-0a9d-4127-b42e-e228b2767edd-kube-api-access-d7q5z\") pod \"maas-api-revoke-sub-62292fe6-zlfk9\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.381717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.381657 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62292fe6-zlfk9\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.381717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.381708 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62292fe6-zlfk9\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.382602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.382577 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62292fe6-zlfk9\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.385512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.385482 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62292fe6-zlfk9\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.399681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.399551 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7q5z\" (UniqueName: \"kubernetes.io/projected/8504a501-0a9d-4127-b42e-e228b2767edd-kube-api-access-d7q5z\") pod \"maas-api-revoke-sub-62292fe6-zlfk9\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.401011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.400918 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:00.467409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.467376 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:35:00.468875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.468846 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:35:00.469001 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.468916 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 01 22:35:00.799185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:00.797731 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9"] Nov 01 22:35:00.806142 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:00.803176 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8504a501_0a9d_4127_b42e_e228b2767edd.slice/crio-57b85253f39a1aedaf0b8ffc0e93390a92ace6f1dee80e913da8baf8412782d2 WatchSource:0}: Error finding container 57b85253f39a1aedaf0b8ffc0e93390a92ace6f1dee80e913da8baf8412782d2: Status 404 returned error can't find the container with id 57b85253f39a1aedaf0b8ffc0e93390a92ace6f1dee80e913da8baf8412782d2 Nov 01 22:35:01.039675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:01.039292 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" event={"ID":"a234993f-449e-4d56-82c5-8bc2282a67fd","Type":"ContainerStarted","Data":"dd42c70b0feda16a153cb16450acbf5358dba85c444c04328685b180c06e07ae"} Nov 01 22:35:01.041735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:01.041706 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" event={"ID":"8504a501-0a9d-4127-b42e-e228b2767edd","Type":"ContainerStarted","Data":"57b85253f39a1aedaf0b8ffc0e93390a92ace6f1dee80e913da8baf8412782d2"} Nov 01 22:35:01.130583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:01.130526 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podStartSLOduration=2.130503471 podStartE2EDuration="2.130503471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:01.103477604 +0000 UTC m=+713.751976842" watchObservedRunningTime="2026-11-01 22:35:01.130503471 +0000 UTC m=+713.779002707" Nov 01 22:35:01.467026 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:01.466942 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:35:01.467026 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:01.466996 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 01 22:35:01.656234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:01.656200 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:35:01.657054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:01.657024 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:35:01.657227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:01.657088 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 01 22:35:02.050868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:02.050828 6287 generic.go:356] "Generic (PLEG): container finished" podID="8504a501-0a9d-4127-b42e-e228b2767edd" containerID="5c48a6453862c73cfa201ffeaec1b6804260c7435e78dc1beb250b116e588980" exitCode=0 Nov 01 22:35:02.051042 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:02.050974 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" event={"ID":"8504a501-0a9d-4127-b42e-e228b2767edd","Type":"ContainerDied","Data":"5c48a6453862c73cfa201ffeaec1b6804260c7435e78dc1beb250b116e588980"} Nov 01 22:35:02.467239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:02.467207 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:35:02.467644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:02.467258 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 01 22:35:02.656436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:02.656402 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:35:02.656607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:02.656465 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 01 22:35:03.440589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.439477 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:03.526714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.526672 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:35:03.527258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.526739 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 01 22:35:03.527699 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.527667 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-openshift-ca\") pod \"8504a501-0a9d-4127-b42e-e228b2767edd\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " Nov 01 22:35:03.527804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.527784 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-xks-ca\") pod \"8504a501-0a9d-4127-b42e-e228b2767edd\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " Nov 01 22:35:03.527894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.527876 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8504a501-0a9d-4127-b42e-e228b2767edd-kube-api-access-d7q5z\" (UniqueName: \"kubernetes.io/projected/8504a501-0a9d-4127-b42e-e228b2767edd-kube-api-access-d7q5z\") pod \"8504a501-0a9d-4127-b42e-e228b2767edd\" (UID: \"8504a501-0a9d-4127-b42e-e228b2767edd\") " Nov 01 22:35:03.528585 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.528561 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-openshift-ca" pod "8504a501-0a9d-4127-b42e-e228b2767edd" (UID: "8504a501-0a9d-4127-b42e-e228b2767edd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:03.531373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.531331 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-xks-ca" pod "8504a501-0a9d-4127-b42e-e228b2767edd" (UID: "8504a501-0a9d-4127-b42e-e228b2767edd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:03.533596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.533567 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8504a501-0a9d-4127-b42e-e228b2767edd-kube-api-access-d7q5z" pod "8504a501-0a9d-4127-b42e-e228b2767edd" (UID: "8504a501-0a9d-4127-b42e-e228b2767edd"). InnerVolumeSpecName "kube-api-access-d7q5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:03.630829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.630729 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7q5z\" (UniqueName: \"kubernetes.io/projected/8504a501-0a9d-4127-b42e-e228b2767edd-kube-api-access-d7q5z\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:03.630829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.630769 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:03.630829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.630786 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8504a501-0a9d-4127-b42e-e228b2767edd-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:03.662504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.661876 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:35:03.662504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:03.661952 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 01 22:35:04.084137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:04.083389 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" event={"ID":"8504a501-0a9d-4127-b42e-e228b2767edd","Type":"ContainerDied","Data":"57b85253f39a1aedaf0b8ffc0e93390a92ace6f1dee80e913da8baf8412782d2"} Nov 01 22:35:04.084137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:04.083456 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57b85253f39a1aedaf0b8ffc0e93390a92ace6f1dee80e913da8baf8412782d2" Nov 01 22:35:04.084137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:04.083529 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9" Nov 01 22:35:04.467522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:04.467313 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:35:04.467522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:04.467376 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 01 22:35:04.655983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:04.655949 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:35:04.656389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:04.656006 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 01 22:35:05.467427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:05.467392 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:35:05.467762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:05.467729 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 01 22:35:05.656145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:05.656092 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:35:05.656520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:05.656189 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 01 22:35:06.466851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:06.466806 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:35:06.467041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:06.466881 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 01 22:35:06.656429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:06.656388 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:35:06.656815 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:06.656469 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 01 22:35:07.475939 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:07.475912 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:35:07.476414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:07.476396 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:35:07.477065 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:07.477044 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:35:07.661105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:07.661076 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:35:07.661602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:07.661578 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:35:07.662462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:07.662442 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:35:08.279884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.279845 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b"] Nov 01 22:35:08.280519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.280504 6287 cpu_manager.go:414] "containerMap: removing container" podUID="8504a501-0a9d-4127-b42e-e228b2767edd" containerName="revoke-keys" Nov 01 22:35:08.280564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.280520 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8504a501-0a9d-4127-b42e-e228b2767edd" containerName="revoke-keys" Nov 01 22:35:08.280668 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.280659 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="8504a501-0a9d-4127-b42e-e228b2767edd" containerName="revoke-keys" Nov 01 22:35:08.285806 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.285780 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.290991 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.290962 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:08.291155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.291031 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:35:08.291824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.291780 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:35:08.291824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.291780 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:35:08.291979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.291836 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b"] Nov 01 22:35:08.416966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.416927 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59688282-rnx8b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.417194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.416993 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59688282-rnx8b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.417194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.417087 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ng4\" (UniqueName: \"kubernetes.io/projected/0a583e01-b0c1-430e-bf60-9db46dd2d27b-kube-api-access-d6ng4\") pod \"maas-api-revoke-sub-59688282-rnx8b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.518210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.518168 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59688282-rnx8b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.518392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.518238 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59688282-rnx8b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.518392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.518276 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ng4\" (UniqueName: \"kubernetes.io/projected/0a583e01-b0c1-430e-bf60-9db46dd2d27b-kube-api-access-d6ng4\") pod \"maas-api-revoke-sub-59688282-rnx8b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.519055 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.519021 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59688282-rnx8b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.520772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.520752 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59688282-rnx8b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.526758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.526736 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ng4\" (UniqueName: \"kubernetes.io/projected/0a583e01-b0c1-430e-bf60-9db46dd2d27b-kube-api-access-d6ng4\") pod \"maas-api-revoke-sub-59688282-rnx8b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.606820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.606740 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:08.907314 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:08.907285 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b"] Nov 01 22:35:09.175787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:09.175749 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" event={"ID":"0a583e01-b0c1-430e-bf60-9db46dd2d27b","Type":"ContainerStarted","Data":"352a8739de5115dceefb21f638ac9823f8e83e64e0e6b308381432d3b9a5eaca"} Nov 01 22:35:10.185710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:10.185620 6287 generic.go:356] "Generic (PLEG): container finished" podID="0a583e01-b0c1-430e-bf60-9db46dd2d27b" containerID="0d120cd1f61e11aebe3ea422ab13e31f14041e50e9b7a91e90285f5b7fe9d041" exitCode=0 Nov 01 22:35:10.186100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:10.185704 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" event={"ID":"0a583e01-b0c1-430e-bf60-9db46dd2d27b","Type":"ContainerDied","Data":"0d120cd1f61e11aebe3ea422ab13e31f14041e50e9b7a91e90285f5b7fe9d041"} Nov 01 22:35:11.374025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.373997 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:11.465588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.465490 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-openshift-ca\") pod \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " Nov 01 22:35:11.465765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.465632 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a583e01-b0c1-430e-bf60-9db46dd2d27b-kube-api-access-d6ng4\" (UniqueName: \"kubernetes.io/projected/0a583e01-b0c1-430e-bf60-9db46dd2d27b-kube-api-access-d6ng4\") pod \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " Nov 01 22:35:11.465765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.465704 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-xks-ca\") pod \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\" (UID: \"0a583e01-b0c1-430e-bf60-9db46dd2d27b\") " Nov 01 22:35:11.465912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.465882 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-openshift-ca" pod "0a583e01-b0c1-430e-bf60-9db46dd2d27b" (UID: "0a583e01-b0c1-430e-bf60-9db46dd2d27b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:11.466309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.466287 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:11.467780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.467755 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-xks-ca" pod "0a583e01-b0c1-430e-bf60-9db46dd2d27b" (UID: "0a583e01-b0c1-430e-bf60-9db46dd2d27b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:11.467990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.467967 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a583e01-b0c1-430e-bf60-9db46dd2d27b-kube-api-access-d6ng4" pod "0a583e01-b0c1-430e-bf60-9db46dd2d27b" (UID: "0a583e01-b0c1-430e-bf60-9db46dd2d27b"). InnerVolumeSpecName "kube-api-access-d6ng4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:11.568238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.568205 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ng4\" (UniqueName: \"kubernetes.io/projected/0a583e01-b0c1-430e-bf60-9db46dd2d27b-kube-api-access-d6ng4\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:11.568238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.568238 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a583e01-b0c1-430e-bf60-9db46dd2d27b-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:11.972434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.972396 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng"] Nov 01 22:35:11.973185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.973161 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0a583e01-b0c1-430e-bf60-9db46dd2d27b" containerName="revoke-keys" Nov 01 22:35:11.973185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.973182 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a583e01-b0c1-430e-bf60-9db46dd2d27b" containerName="revoke-keys" Nov 01 22:35:11.973373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.973344 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a583e01-b0c1-430e-bf60-9db46dd2d27b" containerName="revoke-keys" Nov 01 22:35:11.977143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:11.977097 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.008759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.008725 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng"] Nov 01 22:35:12.076568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.076526 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6d30fde-95sng\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.076737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.076586 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvh9x\" (UniqueName: \"kubernetes.io/projected/e65b68d0-5705-45d7-900f-937a78fe6e7c-kube-api-access-jvh9x\") pod \"maas-api-revoke-sub-b6d30fde-95sng\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.076737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.076665 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6d30fde-95sng\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.177972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.177928 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6d30fde-95sng\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.178189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.177994 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvh9x\" (UniqueName: \"kubernetes.io/projected/e65b68d0-5705-45d7-900f-937a78fe6e7c-kube-api-access-jvh9x\") pod \"maas-api-revoke-sub-b6d30fde-95sng\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.178189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.178036 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6d30fde-95sng\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.178875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.178839 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6d30fde-95sng\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.180749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.180724 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6d30fde-95sng\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.187742 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.187719 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvh9x\" (UniqueName: \"kubernetes.io/projected/e65b68d0-5705-45d7-900f-937a78fe6e7c-kube-api-access-jvh9x\") pod \"maas-api-revoke-sub-b6d30fde-95sng\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.209346 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.209321 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" Nov 01 22:35:12.209499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.209358 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b" event={"ID":"0a583e01-b0c1-430e-bf60-9db46dd2d27b","Type":"ContainerDied","Data":"352a8739de5115dceefb21f638ac9823f8e83e64e0e6b308381432d3b9a5eaca"} Nov 01 22:35:12.209499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.209406 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="352a8739de5115dceefb21f638ac9823f8e83e64e0e6b308381432d3b9a5eaca" Nov 01 22:35:12.291616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.291574 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:12.414478 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.410423 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd"] Nov 01 22:35:12.417453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.417426 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.438463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.437106 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd"] Nov 01 22:35:12.594229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.591580 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a9ab658-rnhcd\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.594229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.591779 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99s9\" (UniqueName: \"kubernetes.io/projected/15180b47-9230-4fe4-9938-92c0183686f5-kube-api-access-c99s9\") pod \"maas-api-revoke-sub-9a9ab658-rnhcd\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.594229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.592001 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a9ab658-rnhcd\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.671560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.671528 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng"] Nov 01 22:35:12.693094 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.693066 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a9ab658-rnhcd\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.693251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.693212 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99s9\" (UniqueName: \"kubernetes.io/projected/15180b47-9230-4fe4-9938-92c0183686f5-kube-api-access-c99s9\") pod \"maas-api-revoke-sub-9a9ab658-rnhcd\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.694315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.693358 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a9ab658-rnhcd\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.705881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.705839 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a9ab658-rnhcd\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.706639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.706250 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a9ab658-rnhcd\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.710980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.710947 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99s9\" (UniqueName: \"kubernetes.io/projected/15180b47-9230-4fe4-9938-92c0183686f5-kube-api-access-c99s9\") pod \"maas-api-revoke-sub-9a9ab658-rnhcd\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:12.756231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:12.753484 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:13.173787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:13.173081 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd"] Nov 01 22:35:13.237136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:13.235027 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" event={"ID":"e65b68d0-5705-45d7-900f-937a78fe6e7c","Type":"ContainerStarted","Data":"a98031ebe6595b421379c662b92533dd5c612bab357095e549f9e63186c6c1b0"} Nov 01 22:35:13.237136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:13.235099 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" event={"ID":"e65b68d0-5705-45d7-900f-937a78fe6e7c","Type":"ContainerStarted","Data":"4b502b35671bb9afa242626711a7dcbacbb79223fc6b320835bf5e2df8ccc4af"} Nov 01 22:35:13.242179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:13.237904 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" event={"ID":"15180b47-9230-4fe4-9938-92c0183686f5","Type":"ContainerStarted","Data":"ce2bb0705d8a52605f50097e09a4d9b29d3d2f336a70b8e2adc5050f669df9b9"} Nov 01 22:35:13.260138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:13.254216 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" podStartSLOduration=2.254195734 podStartE2EDuration="2.254195734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:13.252584619 +0000 UTC m=+725.901083854" watchObservedRunningTime="2026-11-01 22:35:13.254195734 +0000 UTC m=+725.902694974" Nov 01 22:35:14.266147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.258785 6287 generic.go:356] "Generic (PLEG): container finished" podID="e65b68d0-5705-45d7-900f-937a78fe6e7c" containerID="a98031ebe6595b421379c662b92533dd5c612bab357095e549f9e63186c6c1b0" exitCode=0 Nov 01 22:35:14.266147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.259056 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" event={"ID":"e65b68d0-5705-45d7-900f-937a78fe6e7c","Type":"ContainerDied","Data":"a98031ebe6595b421379c662b92533dd5c612bab357095e549f9e63186c6c1b0"} Nov 01 22:35:14.266147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.261768 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" event={"ID":"15180b47-9230-4fe4-9938-92c0183686f5","Type":"ContainerStarted","Data":"c126d130ce2c7b9d5e63445550210a465c22f830563eab1279144889c58e24e4"} Nov 01 22:35:14.366132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.363394 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz"] Nov 01 22:35:14.492337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.490637 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" podStartSLOduration=2.490613048 podStartE2EDuration="2.490613048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:14.303900083 +0000 UTC m=+726.952399318" watchObservedRunningTime="2026-11-01 22:35:14.490613048 +0000 UTC m=+727.139112284" Nov 01 22:35:14.738964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.737211 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:14.788138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.781636 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz"] Nov 01 22:35:14.788138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.781791 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx"] Nov 01 22:35:14.788138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.782244 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.800396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.800369 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx"] Nov 01 22:35:14.800674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.800647 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc"] Nov 01 22:35:14.812438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.812410 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.817660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.817635 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:14.824138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.822372 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-b28a40-openshift-default-dockercfg-mwllr\"" Nov 01 22:35:14.857167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.839068 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:14.895816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.895779 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.896009 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.895990 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.896207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.896188 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:14.896322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.896307 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.896437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.896421 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.896542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.896528 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.896692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.896676 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:14.896820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.896804 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:14.896928 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.896914 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:14.897057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.897041 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9k9w\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-kube-api-access-n9k9w\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:14.905229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.905205 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.905435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.905414 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.905579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.905560 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.905700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.905680 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.905818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.905800 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.905941 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.905925 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:14.906056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.904599 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc"] Nov 01 22:35:14.906266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.906040 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:14.906410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.906391 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.906538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.906518 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.906673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.906655 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.906844 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.906826 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.906982 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.906965 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94nqm\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-kube-api-access-94nqm\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.907129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.907095 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.907279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.907262 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:14.907402 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.907387 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:14.907512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.907497 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6kmw\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-kube-api-access-g6kmw\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:14.907805 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:14.907782 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.012900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.012811 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.024541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.024510 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.025794 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.025772 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9k9w\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-kube-api-access-n9k9w\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.025971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.025951 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.026341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.026142 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.027579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.027560 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.027714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.027697 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.027839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.027824 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.032203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.032181 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.033149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.033128 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.034915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.030809 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istiod-ca-cert\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.035434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.027502 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.035604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.031347 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-data\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.035752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.027204 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-credential-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.035906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.032595 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.036055 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.033900 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.036211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.025708 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.036338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.034849 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.036486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.036470 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.037015 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.036997 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.037677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.036922 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.038360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.037843 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.038700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.038522 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94nqm\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-kube-api-access-94nqm\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.039873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.039396 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.040161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.039774 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.040161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.038223 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-certs\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.040569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.040548 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.041073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.041051 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.061024 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6kmw\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-kube-api-access-g6kmw\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.061078 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.061200 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.061249 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.061292 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.061317 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.061343 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.061374 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.061431 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.062732 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.059499 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-token\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.063326 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94nqm\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-kube-api-access-94nqm\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.063613 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.091203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.064429 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.097073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.094075 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.097073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.096391 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-socket\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.097073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.097027 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.109650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.109626 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.130134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.125943 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.148475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.147850 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.148924 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.148871 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-envoy\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.150140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.149667 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-podinfo\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.150314 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.150265 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6kmw\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-kube-api-access-g6kmw\") pod \"e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.158530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.158508 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9k9w\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-kube-api-access-n9k9w\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.164474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.164452 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.167639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.167616 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.222360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.219847 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:15.241601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.241558 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:15.280588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.280499 6287 generic.go:356] "Generic (PLEG): container finished" podID="15180b47-9230-4fe4-9938-92c0183686f5" containerID="c126d130ce2c7b9d5e63445550210a465c22f830563eab1279144889c58e24e4" exitCode=0 Nov 01 22:35:15.282031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.281995 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" event={"ID":"15180b47-9230-4fe4-9938-92c0183686f5","Type":"ContainerDied","Data":"c126d130ce2c7b9d5e63445550210a465c22f830563eab1279144889c58e24e4"} Nov 01 22:35:15.284177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.283495 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:15.677241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.668648 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz"] Nov 01 22:35:15.729386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.700028 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:35:15.729386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.700202 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:35:15.729386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.700292 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:35:15.737771 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:35:15.737736 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:35:15.739783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.739742 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:35:15.742280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.742245 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:35:15.742512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.742489 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:35:15.814090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.814054 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc"] Nov 01 22:35:15.932871 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:15.932840 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11f70a0_4ce1_48ff_97fa_ec62f6e0ffab.slice/crio-5d68f90f739202c3e5bfa1b8cc2000f680fc27c59dbf422c009a07d7958accef WatchSource:0}: Error finding container 5d68f90f739202c3e5bfa1b8cc2000f680fc27c59dbf422c009a07d7958accef: Status 404 returned error can't find the container with id 5d68f90f739202c3e5bfa1b8cc2000f680fc27c59dbf422c009a07d7958accef Nov 01 22:35:15.986530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:15.986498 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:16.011519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.003599 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:35:16.011519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.003685 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:35:16.011519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.003723 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:35:16.034961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.034927 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-openshift-ca\") pod \"e65b68d0-5705-45d7-900f-937a78fe6e7c\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " Nov 01 22:35:16.040083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.040020 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65b68d0-5705-45d7-900f-937a78fe6e7c-kube-api-access-jvh9x\" (UniqueName: \"kubernetes.io/projected/e65b68d0-5705-45d7-900f-937a78fe6e7c-kube-api-access-jvh9x\") pod \"e65b68d0-5705-45d7-900f-937a78fe6e7c\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " Nov 01 22:35:16.040961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.040939 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-xks-ca\") pod \"e65b68d0-5705-45d7-900f-937a78fe6e7c\" (UID: \"e65b68d0-5705-45d7-900f-937a78fe6e7c\") " Nov 01 22:35:16.041552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.038346 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-openshift-ca" pod "e65b68d0-5705-45d7-900f-937a78fe6e7c" (UID: "e65b68d0-5705-45d7-900f-937a78fe6e7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:16.043138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.043094 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65b68d0-5705-45d7-900f-937a78fe6e7c-kube-api-access-jvh9x" pod "e65b68d0-5705-45d7-900f-937a78fe6e7c" (UID: "e65b68d0-5705-45d7-900f-937a78fe6e7c"). InnerVolumeSpecName "kube-api-access-jvh9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:16.044837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.044817 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-xks-ca" pod "e65b68d0-5705-45d7-900f-937a78fe6e7c" (UID: "e65b68d0-5705-45d7-900f-937a78fe6e7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:16.047957 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.047878 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx"] Nov 01 22:35:16.161167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.148502 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:16.161167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.148539 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvh9x\" (UniqueName: \"kubernetes.io/projected/e65b68d0-5705-45d7-900f-937a78fe6e7c-kube-api-access-jvh9x\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:16.161167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.148551 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e65b68d0-5705-45d7-900f-937a78fe6e7c-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:16.304347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.304250 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" event={"ID":"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f","Type":"ContainerStarted","Data":"68951c35f590522a12708b2d25dbdfffd3cca373f196a4efb49f22a4cdad63f9"} Nov 01 22:35:16.304967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.304943 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" event={"ID":"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f","Type":"ContainerStarted","Data":"f250cc7491e398d0cb259a542ac19b762928736247eee9227679df1d51123522"} Nov 01 22:35:16.310617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.308671 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" event={"ID":"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab","Type":"ContainerStarted","Data":"34445fc5ff0916934f6a904bd594fef5d16b86514fd8865061e9c61958bdfe63"} Nov 01 22:35:16.310617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.308722 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" event={"ID":"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab","Type":"ContainerStarted","Data":"5d68f90f739202c3e5bfa1b8cc2000f680fc27c59dbf422c009a07d7958accef"} Nov 01 22:35:16.313189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.313163 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" event={"ID":"6d0f5f78-2e7f-4b2b-92a1-c893385f265b","Type":"ContainerStarted","Data":"be353f7f6586d91f1e3af2e668877affd3a3c5fb5fb4c5e008af842b9fcfbe8c"} Nov 01 22:35:16.313308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.313211 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" event={"ID":"6d0f5f78-2e7f-4b2b-92a1-c893385f265b","Type":"ContainerStarted","Data":"a3c169d838247125f06dd8479a190468a239baadf2386d4be3e253c8d0f08c77"} Nov 01 22:35:16.315369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.315343 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" Nov 01 22:35:16.319701 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.319639 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng" event={"ID":"e65b68d0-5705-45d7-900f-937a78fe6e7c","Type":"ContainerDied","Data":"4b502b35671bb9afa242626711a7dcbacbb79223fc6b320835bf5e2df8ccc4af"} Nov 01 22:35:16.319701 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.319683 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b502b35671bb9afa242626711a7dcbacbb79223fc6b320835bf5e2df8ccc4af" Nov 01 22:35:16.383309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.383244 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podStartSLOduration=2.383224596 podStartE2EDuration="2.383224596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:16.373570482 +0000 UTC m=+729.022069721" watchObservedRunningTime="2026-11-01 22:35:16.383224596 +0000 UTC m=+729.031723833" Nov 01 22:35:16.435126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.435051 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podStartSLOduration=2.435033361 podStartE2EDuration="2.435033361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:16.426029625 +0000 UTC m=+729.074528860" watchObservedRunningTime="2026-11-01 22:35:16.435033361 +0000 UTC m=+729.083532597" Nov 01 22:35:16.540096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.520548 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" podStartSLOduration=3.520526127 podStartE2EDuration="3.520526127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:16.516370169 +0000 UTC m=+729.164869406" watchObservedRunningTime="2026-11-01 22:35:16.520526127 +0000 UTC m=+729.169025363" Nov 01 22:35:16.714547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.714294 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:16.762642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.762593 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-openshift-ca\") pod \"15180b47-9230-4fe4-9938-92c0183686f5\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " Nov 01 22:35:16.762977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.762950 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-xks-ca\") pod \"15180b47-9230-4fe4-9938-92c0183686f5\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " Nov 01 22:35:16.763354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.763334 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15180b47-9230-4fe4-9938-92c0183686f5-kube-api-access-c99s9\" (UniqueName: \"kubernetes.io/projected/15180b47-9230-4fe4-9938-92c0183686f5-kube-api-access-c99s9\") pod \"15180b47-9230-4fe4-9938-92c0183686f5\" (UID: \"15180b47-9230-4fe4-9938-92c0183686f5\") " Nov 01 22:35:16.765220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.765190 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-openshift-ca" pod "15180b47-9230-4fe4-9938-92c0183686f5" (UID: "15180b47-9230-4fe4-9938-92c0183686f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:16.807142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.807086 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15180b47-9230-4fe4-9938-92c0183686f5-kube-api-access-c99s9" pod "15180b47-9230-4fe4-9938-92c0183686f5" (UID: "15180b47-9230-4fe4-9938-92c0183686f5"). InnerVolumeSpecName "kube-api-access-c99s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:16.831165 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.820303 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-xks-ca" pod "15180b47-9230-4fe4-9938-92c0183686f5" (UID: "15180b47-9230-4fe4-9938-92c0183686f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:16.868075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.867997 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99s9\" (UniqueName: \"kubernetes.io/projected/15180b47-9230-4fe4-9938-92c0183686f5-kube-api-access-c99s9\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:16.868075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.868033 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:16.868075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:16.868049 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15180b47-9230-4fe4-9938-92c0183686f5-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:17.245046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.244178 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:17.245046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.244316 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 22:35:17.245046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.244360 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 22:35:17.256545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.254253 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:17.256545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.256233 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:35:17.256545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.256293 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 22:35:17.333187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.311190 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:17.333187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.311388 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:35:17.333187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.311441 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 22:35:17.432450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.432411 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" Nov 01 22:35:17.444310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.444276 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd" event={"ID":"15180b47-9230-4fe4-9938-92c0183686f5","Type":"ContainerDied","Data":"ce2bb0705d8a52605f50097e09a4d9b29d3d2f336a70b8e2adc5050f669df9b9"} Nov 01 22:35:17.444556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.444535 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce2bb0705d8a52605f50097e09a4d9b29d3d2f336a70b8e2adc5050f669df9b9" Nov 01 22:35:17.959286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.959138 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969"] Nov 01 22:35:17.960552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.960507 6287 cpu_manager.go:414] "containerMap: removing container" podUID="15180b47-9230-4fe4-9938-92c0183686f5" containerName="revoke-keys" Nov 01 22:35:17.960747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.960728 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15180b47-9230-4fe4-9938-92c0183686f5" containerName="revoke-keys" Nov 01 22:35:17.960882 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.960867 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e65b68d0-5705-45d7-900f-937a78fe6e7c" containerName="revoke-keys" Nov 01 22:35:17.960992 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.960978 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65b68d0-5705-45d7-900f-937a78fe6e7c" containerName="revoke-keys" Nov 01 22:35:17.961376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.961343 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="15180b47-9230-4fe4-9938-92c0183686f5" containerName="revoke-keys" Nov 01 22:35:17.961529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.961514 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65b68d0-5705-45d7-900f-937a78fe6e7c" containerName="revoke-keys" Nov 01 22:35:17.966545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.966498 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:17.989138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:17.967983 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969"] Nov 01 22:35:18.104137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.098188 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:35:18.104137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.098310 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:35:18.104137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.098392 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:18.104137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.098474 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:35:18.213221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.207368 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlhmn\" (UniqueName: \"kubernetes.io/projected/ff4bba59-43df-4446-a6dd-f7891cc377fc-kube-api-access-jlhmn\") pod \"maas-api-revoke-sub-d53d1339-nj969\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:18.213221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.207532 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53d1339-nj969\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:18.213221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.207635 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53d1339-nj969\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:18.232865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.223459 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.232865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.227565 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 22:35:18.255137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.249227 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.255137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.249287 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 22:35:18.291916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.291883 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.292195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.292166 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 22:35:18.313388 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.313350 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53d1339-nj969\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:18.319284 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.319255 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53d1339-nj969\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:18.319653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.319624 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlhmn\" (UniqueName: \"kubernetes.io/projected/ff4bba59-43df-4446-a6dd-f7891cc377fc-kube-api-access-jlhmn\") pod \"maas-api-revoke-sub-d53d1339-nj969\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:18.320959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.320926 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53d1339-nj969\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:18.393682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.393645 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlhmn\" (UniqueName: \"kubernetes.io/projected/ff4bba59-43df-4446-a6dd-f7891cc377fc-kube-api-access-jlhmn\") pod \"maas-api-revoke-sub-d53d1339-nj969\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:18.447140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.436059 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53d1339-nj969\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:18.565718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:18.565691 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:19.245196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:19.230542 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 22:35:19.245196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:19.230606 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 22:35:19.268136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:19.267653 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:35:19.268136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:19.267709 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 22:35:19.290135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:19.277282 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969"] Nov 01 22:35:19.303137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:19.302226 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:35:19.303137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:19.302292 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 22:35:19.559162 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:19.557704 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" event={"ID":"ff4bba59-43df-4446-a6dd-f7891cc377fc","Type":"ContainerStarted","Data":"065cb8b7a04fac8adbe0638ef7e9a5bbc1fb0354e60b9b6632ff22d1df229dbb"} Nov 01 22:35:20.061785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.061752 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6"] Nov 01 22:35:20.069340 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.069042 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.175264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.163512 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2gd8\" (UniqueName: \"kubernetes.io/projected/faa21f90-4e1e-432e-b0da-4d1108390782-kube-api-access-c2gd8\") pod \"maas-api-revoke-sub-cc42a633-8lcs6\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.175264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.163720 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc42a633-8lcs6\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.175264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.163869 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc42a633-8lcs6\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.241623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.241587 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 22:35:20.247488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.242411 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:35:20.260039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.259991 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 22:35:20.260611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.241867 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 22:35:20.269022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.268998 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6"] Nov 01 22:35:20.330150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.318185 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:35:20.330150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.318247 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 22:35:20.347346 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.347304 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc42a633-8lcs6\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.351004 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.350014 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2gd8\" (UniqueName: \"kubernetes.io/projected/faa21f90-4e1e-432e-b0da-4d1108390782-kube-api-access-c2gd8\") pod \"maas-api-revoke-sub-cc42a633-8lcs6\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.351484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.351446 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc42a633-8lcs6\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.354966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.354919 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc42a633-8lcs6\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.357187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.357168 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc42a633-8lcs6\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.363797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.363778 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2gd8\" (UniqueName: \"kubernetes.io/projected/faa21f90-4e1e-432e-b0da-4d1108390782-kube-api-access-c2gd8\") pod \"maas-api-revoke-sub-cc42a633-8lcs6\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.599135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.598375 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" event={"ID":"ff4bba59-43df-4446-a6dd-f7891cc377fc","Type":"ContainerStarted","Data":"f12c0a2171b504f9a01db45c7defc65171e01d86ee66357cbb16459f30f9fa73"} Nov 01 22:35:20.626310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.621411 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:20.730139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:20.722453 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" podStartSLOduration=3.722429447 podStartE2EDuration="3.722429447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:20.652452667 +0000 UTC m=+733.300951888" watchObservedRunningTime="2026-11-01 22:35:20.722429447 +0000 UTC m=+733.370928683" Nov 01 22:35:21.267800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:21.267265 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 22:35:21.267800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:21.267326 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 22:35:21.269602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:21.267265 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:35:21.269602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:21.269107 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 22:35:21.380444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:21.363563 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:35:21.380444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:21.363616 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 22:35:21.826700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:21.826658 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96"] Nov 01 22:35:22.084213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.083347 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96"] Nov 01 22:35:22.084213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.083416 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6"] Nov 01 22:35:22.084213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.083532 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.160057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.157183 6287 generic.go:356] "Generic (PLEG): container finished" podID="ff4bba59-43df-4446-a6dd-f7891cc377fc" containerID="f12c0a2171b504f9a01db45c7defc65171e01d86ee66357cbb16459f30f9fa73" exitCode=0 Nov 01 22:35:22.160057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.157472 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" event={"ID":"ff4bba59-43df-4446-a6dd-f7891cc377fc","Type":"ContainerDied","Data":"f12c0a2171b504f9a01db45c7defc65171e01d86ee66357cbb16459f30f9fa73"} Nov 01 22:35:22.269315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.269264 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87e818e-gbr96\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.269464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.269336 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwj2\" (UniqueName: \"kubernetes.io/projected/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-kube-api-access-rhwj2\") pod \"maas-api-revoke-sub-c87e818e-gbr96\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.269728 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.269709 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87e818e-gbr96\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.269967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.269943 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.270047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.270019 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 22:35:22.354463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.352289 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.354463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.352373 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 22:35:22.354463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.352826 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.354463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.352888 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 22:35:22.395511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.395440 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87e818e-gbr96\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.395682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.395542 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87e818e-gbr96\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.395682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.395585 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwj2\" (UniqueName: \"kubernetes.io/projected/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-kube-api-access-rhwj2\") pod \"maas-api-revoke-sub-c87e818e-gbr96\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.417016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.416982 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87e818e-gbr96\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.458643 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.458604 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87e818e-gbr96\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.461276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.461241 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwj2\" (UniqueName: \"kubernetes.io/projected/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-kube-api-access-rhwj2\") pod \"maas-api-revoke-sub-c87e818e-gbr96\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.467997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.467974 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:22.916016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:22.915986 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96"] Nov 01 22:35:22.916489 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:22.916464 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bab43fc_43ee_4869_9cc7_3fcded2f4c55.slice/crio-22ccee77c5a376c6cf10c4904573e1bd948eb06f2d7b9952dcd934c900597c74 WatchSource:0}: Error finding container 22ccee77c5a376c6cf10c4904573e1bd948eb06f2d7b9952dcd934c900597c74: Status 404 returned error can't find the container with id 22ccee77c5a376c6cf10c4904573e1bd948eb06f2d7b9952dcd934c900597c74 Nov 01 22:35:23.168970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.168933 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" event={"ID":"faa21f90-4e1e-432e-b0da-4d1108390782","Type":"ContainerStarted","Data":"a2110b3435b241660485d4b002a6615ea2fcea030d1e5a1cb52e7c25453c2ec5"} Nov 01 22:35:23.169187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.169001 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" event={"ID":"faa21f90-4e1e-432e-b0da-4d1108390782","Type":"ContainerStarted","Data":"b11a2a161fd61e0b0ed075b0802452d44ce7f4b0aa81dde63714515d68e2d398"} Nov 01 22:35:23.170466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.170434 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" event={"ID":"2bab43fc-43ee-4869-9cc7-3fcded2f4c55","Type":"ContainerStarted","Data":"22ccee77c5a376c6cf10c4904573e1bd948eb06f2d7b9952dcd934c900597c74"} Nov 01 22:35:23.214883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.214747 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" podStartSLOduration=3.214726223 podStartE2EDuration="3.214726223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:23.208465268 +0000 UTC m=+735.856964505" watchObservedRunningTime="2026-11-01 22:35:23.214726223 +0000 UTC m=+735.863225461" Nov 01 22:35:23.223994 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.223955 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.224163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.224021 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 01 22:35:23.242528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.242498 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.242742 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.242715 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 22:35:23.284491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.284456 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.284649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.284520 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 22:35:23.402658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.402626 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:23.453454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.453423 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-openshift-ca\") pod \"ff4bba59-43df-4446-a6dd-f7891cc377fc\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " Nov 01 22:35:23.453655 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.453509 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-xks-ca\") pod \"ff4bba59-43df-4446-a6dd-f7891cc377fc\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " Nov 01 22:35:23.453655 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.453625 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4bba59-43df-4446-a6dd-f7891cc377fc-kube-api-access-jlhmn\" (UniqueName: \"kubernetes.io/projected/ff4bba59-43df-4446-a6dd-f7891cc377fc-kube-api-access-jlhmn\") pod \"ff4bba59-43df-4446-a6dd-f7891cc377fc\" (UID: \"ff4bba59-43df-4446-a6dd-f7891cc377fc\") " Nov 01 22:35:23.454189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.454161 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-openshift-ca" pod "ff4bba59-43df-4446-a6dd-f7891cc377fc" (UID: "ff4bba59-43df-4446-a6dd-f7891cc377fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:23.456961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.456933 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-xks-ca" pod "ff4bba59-43df-4446-a6dd-f7891cc377fc" (UID: "ff4bba59-43df-4446-a6dd-f7891cc377fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:23.457215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.457193 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4bba59-43df-4446-a6dd-f7891cc377fc-kube-api-access-jlhmn" pod "ff4bba59-43df-4446-a6dd-f7891cc377fc" (UID: "ff4bba59-43df-4446-a6dd-f7891cc377fc"). InnerVolumeSpecName "kube-api-access-jlhmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:23.555669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.555628 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:23.555669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.555659 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4bba59-43df-4446-a6dd-f7891cc377fc-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:23.555669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:23.555673 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlhmn\" (UniqueName: \"kubernetes.io/projected/ff4bba59-43df-4446-a6dd-f7891cc377fc-kube-api-access-jlhmn\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:24.189636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.189601 6287 generic.go:356] "Generic (PLEG): container finished" podID="faa21f90-4e1e-432e-b0da-4d1108390782" containerID="a2110b3435b241660485d4b002a6615ea2fcea030d1e5a1cb52e7c25453c2ec5" exitCode=0 Nov 01 22:35:24.190267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.189694 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" event={"ID":"faa21f90-4e1e-432e-b0da-4d1108390782","Type":"ContainerDied","Data":"a2110b3435b241660485d4b002a6615ea2fcea030d1e5a1cb52e7c25453c2ec5"} Nov 01 22:35:24.192780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.192484 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" Nov 01 22:35:24.192780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.192616 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969" event={"ID":"ff4bba59-43df-4446-a6dd-f7891cc377fc","Type":"ContainerDied","Data":"065cb8b7a04fac8adbe0638ef7e9a5bbc1fb0354e60b9b6632ff22d1df229dbb"} Nov 01 22:35:24.192780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.192655 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="065cb8b7a04fac8adbe0638ef7e9a5bbc1fb0354e60b9b6632ff22d1df229dbb" Nov 01 22:35:24.202176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.201290 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" event={"ID":"2bab43fc-43ee-4869-9cc7-3fcded2f4c55","Type":"ContainerStarted","Data":"ed382ced0491387b641688e778629b34f86c2a6a432b47b4bf19184fce8cb3b9"} Nov 01 22:35:24.240688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.240620 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" podStartSLOduration=3.240600981 podStartE2EDuration="3.240600981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:24.234447872 +0000 UTC m=+736.882947108" watchObservedRunningTime="2026-11-01 22:35:24.240600981 +0000 UTC m=+736.889100216" Nov 01 22:35:24.267915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.267841 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:24.274563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.274141 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:24.280621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.277909 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:35:24.308916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.308891 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:24.309578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.309561 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:24.311437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.311418 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:35:24.357150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.354471 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:24.357150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.356102 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:24.359951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:24.358892 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:35:25.222393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.222349 6287 generic.go:356] "Generic (PLEG): container finished" podID="2bab43fc-43ee-4869-9cc7-3fcded2f4c55" containerID="ed382ced0491387b641688e778629b34f86c2a6a432b47b4bf19184fce8cb3b9" exitCode=0 Nov 01 22:35:25.222846 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.222425 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" event={"ID":"2bab43fc-43ee-4869-9cc7-3fcded2f4c55","Type":"ContainerDied","Data":"ed382ced0491387b641688e778629b34f86c2a6a432b47b4bf19184fce8cb3b9"} Nov 01 22:35:25.404033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.404002 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:25.524291 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.524263 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-openshift-ca\") pod \"faa21f90-4e1e-432e-b0da-4d1108390782\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " Nov 01 22:35:25.524438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.524398 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa21f90-4e1e-432e-b0da-4d1108390782-kube-api-access-c2gd8\" (UniqueName: \"kubernetes.io/projected/faa21f90-4e1e-432e-b0da-4d1108390782-kube-api-access-c2gd8\") pod \"faa21f90-4e1e-432e-b0da-4d1108390782\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " Nov 01 22:35:25.524488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.524439 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-xks-ca\") pod \"faa21f90-4e1e-432e-b0da-4d1108390782\" (UID: \"faa21f90-4e1e-432e-b0da-4d1108390782\") " Nov 01 22:35:25.524610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.524585 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-openshift-ca" pod "faa21f90-4e1e-432e-b0da-4d1108390782" (UID: "faa21f90-4e1e-432e-b0da-4d1108390782"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:25.524996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.524975 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:25.526572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.526552 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa21f90-4e1e-432e-b0da-4d1108390782-kube-api-access-c2gd8" pod "faa21f90-4e1e-432e-b0da-4d1108390782" (UID: "faa21f90-4e1e-432e-b0da-4d1108390782"). InnerVolumeSpecName "kube-api-access-c2gd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:25.526636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.526557 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-xks-ca" pod "faa21f90-4e1e-432e-b0da-4d1108390782" (UID: "faa21f90-4e1e-432e-b0da-4d1108390782"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:25.626849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.626821 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2gd8\" (UniqueName: \"kubernetes.io/projected/faa21f90-4e1e-432e-b0da-4d1108390782-kube-api-access-c2gd8\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:25.626849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:25.626848 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faa21f90-4e1e-432e-b0da-4d1108390782-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:26.204022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.203990 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv"] Nov 01 22:35:26.204639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.204625 6287 cpu_manager.go:414] "containerMap: removing container" podUID="faa21f90-4e1e-432e-b0da-4d1108390782" containerName="revoke-keys" Nov 01 22:35:26.204686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.204640 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa21f90-4e1e-432e-b0da-4d1108390782" containerName="revoke-keys" Nov 01 22:35:26.204686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.204657 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ff4bba59-43df-4446-a6dd-f7891cc377fc" containerName="revoke-keys" Nov 01 22:35:26.204686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.204662 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4bba59-43df-4446-a6dd-f7891cc377fc" containerName="revoke-keys" Nov 01 22:35:26.204783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.204774 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4bba59-43df-4446-a6dd-f7891cc377fc" containerName="revoke-keys" Nov 01 22:35:26.204815 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.204793 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa21f90-4e1e-432e-b0da-4d1108390782" containerName="revoke-keys" Nov 01 22:35:26.208699 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.208682 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.217069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.217042 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv"] Nov 01 22:35:26.242156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.239790 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" Nov 01 22:35:26.242156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.239814 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6" event={"ID":"faa21f90-4e1e-432e-b0da-4d1108390782","Type":"ContainerDied","Data":"b11a2a161fd61e0b0ed075b0802452d44ce7f4b0aa81dde63714515d68e2d398"} Nov 01 22:35:26.242156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.239867 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b11a2a161fd61e0b0ed075b0802452d44ce7f4b0aa81dde63714515d68e2d398" Nov 01 22:35:26.336333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.336304 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkqvr\" (UniqueName: \"kubernetes.io/projected/f8b26bb0-edc8-43da-9237-5ba2598af0fd-kube-api-access-hkqvr\") pod \"maas-api-revoke-sub-360ed6d2-25lxv\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.336849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.336660 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360ed6d2-25lxv\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.336849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.336812 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360ed6d2-25lxv\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.425893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.425866 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:26.437996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.437972 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360ed6d2-25lxv\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.438099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.438065 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkqvr\" (UniqueName: \"kubernetes.io/projected/f8b26bb0-edc8-43da-9237-5ba2598af0fd-kube-api-access-hkqvr\") pod \"maas-api-revoke-sub-360ed6d2-25lxv\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.438254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.438241 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360ed6d2-25lxv\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.439014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.438988 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360ed6d2-25lxv\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.440401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.440381 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360ed6d2-25lxv\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.449804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.449778 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkqvr\" (UniqueName: \"kubernetes.io/projected/f8b26bb0-edc8-43da-9237-5ba2598af0fd-kube-api-access-hkqvr\") pod \"maas-api-revoke-sub-360ed6d2-25lxv\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.526960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.526924 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:26.538802 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.538770 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-kube-api-access-rhwj2\" (UniqueName: \"kubernetes.io/projected/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-kube-api-access-rhwj2\") pod \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " Nov 01 22:35:26.540041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.538920 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-openshift-ca\") pod \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " Nov 01 22:35:26.540041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.539135 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-xks-ca\") pod \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\" (UID: \"2bab43fc-43ee-4869-9cc7-3fcded2f4c55\") " Nov 01 22:35:26.540041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.539649 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-openshift-ca" pod "2bab43fc-43ee-4869-9cc7-3fcded2f4c55" (UID: "2bab43fc-43ee-4869-9cc7-3fcded2f4c55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:26.540041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.539842 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:26.542408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.542377 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-xks-ca" pod "2bab43fc-43ee-4869-9cc7-3fcded2f4c55" (UID: "2bab43fc-43ee-4869-9cc7-3fcded2f4c55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:26.542821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.542788 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-kube-api-access-rhwj2" pod "2bab43fc-43ee-4869-9cc7-3fcded2f4c55" (UID: "2bab43fc-43ee-4869-9cc7-3fcded2f4c55"). InnerVolumeSpecName "kube-api-access-rhwj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:26.642030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.641989 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhwj2\" (UniqueName: \"kubernetes.io/projected/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-kube-api-access-rhwj2\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:26.642214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.642100 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bab43fc-43ee-4869-9cc7-3fcded2f4c55-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:26.828835 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:26.828787 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b26bb0_edc8_43da_9237_5ba2598af0fd.slice/crio-53cab7b5be5d24a96c418b0e831a6f754916b1669dfd3ccb5e4c4c5102a8671b WatchSource:0}: Error finding container 53cab7b5be5d24a96c418b0e831a6f754916b1669dfd3ccb5e4c4c5102a8671b: Status 404 returned error can't find the container with id 53cab7b5be5d24a96c418b0e831a6f754916b1669dfd3ccb5e4c4c5102a8671b Nov 01 22:35:26.829804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:26.829778 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv"] Nov 01 22:35:27.251316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:27.251276 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" event={"ID":"f8b26bb0-edc8-43da-9237-5ba2598af0fd","Type":"ContainerStarted","Data":"e873a25a98f1f4af192f47c019b1b8f2616bafbb1f47b7c84a6cf06c98a53339"} Nov 01 22:35:27.251753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:27.251391 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" event={"ID":"f8b26bb0-edc8-43da-9237-5ba2598af0fd","Type":"ContainerStarted","Data":"53cab7b5be5d24a96c418b0e831a6f754916b1669dfd3ccb5e4c4c5102a8671b"} Nov 01 22:35:27.253080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:27.253050 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" event={"ID":"2bab43fc-43ee-4869-9cc7-3fcded2f4c55","Type":"ContainerDied","Data":"22ccee77c5a376c6cf10c4904573e1bd948eb06f2d7b9952dcd934c900597c74"} Nov 01 22:35:27.253269 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:27.253122 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ccee77c5a376c6cf10c4904573e1bd948eb06f2d7b9952dcd934c900597c74" Nov 01 22:35:27.253269 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:27.253058 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96" Nov 01 22:35:27.280694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:27.280646 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" podStartSLOduration=1.280633625 podStartE2EDuration="1.280633625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:27.277049683 +0000 UTC m=+739.925548919" watchObservedRunningTime="2026-11-01 22:35:27.280633625 +0000 UTC m=+739.929132898" Nov 01 22:35:28.076431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.076398 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9"] Nov 01 22:35:28.077134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.077100 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2bab43fc-43ee-4869-9cc7-3fcded2f4c55" containerName="revoke-keys" Nov 01 22:35:28.077134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.077134 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bab43fc-43ee-4869-9cc7-3fcded2f4c55" containerName="revoke-keys" Nov 01 22:35:28.077345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.077330 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bab43fc-43ee-4869-9cc7-3fcded2f4c55" containerName="revoke-keys" Nov 01 22:35:28.081102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.081084 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.090374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.090352 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9"] Nov 01 22:35:28.178503 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.178472 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzjx\" (UniqueName: \"kubernetes.io/projected/b004e7b4-0d12-4916-95ff-16dc7ddc407d-kube-api-access-xwzjx\") pod \"maas-api-revoke-sub-b37291d3-ghhr9\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.178684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.178522 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37291d3-ghhr9\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.178684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.178642 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37291d3-ghhr9\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.274915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.274872 6287 generic.go:356] "Generic (PLEG): container finished" podID="f8b26bb0-edc8-43da-9237-5ba2598af0fd" containerID="e873a25a98f1f4af192f47c019b1b8f2616bafbb1f47b7c84a6cf06c98a53339" exitCode=0 Nov 01 22:35:28.275397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.274919 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" event={"ID":"f8b26bb0-edc8-43da-9237-5ba2598af0fd","Type":"ContainerDied","Data":"e873a25a98f1f4af192f47c019b1b8f2616bafbb1f47b7c84a6cf06c98a53339"} Nov 01 22:35:28.279863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.279835 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37291d3-ghhr9\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.280013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.279895 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37291d3-ghhr9\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.280256 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.280234 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzjx\" (UniqueName: \"kubernetes.io/projected/b004e7b4-0d12-4916-95ff-16dc7ddc407d-kube-api-access-xwzjx\") pod \"maas-api-revoke-sub-b37291d3-ghhr9\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.280761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.280737 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37291d3-ghhr9\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.282401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.282380 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37291d3-ghhr9\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.293280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.293260 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzjx\" (UniqueName: \"kubernetes.io/projected/b004e7b4-0d12-4916-95ff-16dc7ddc407d-kube-api-access-xwzjx\") pod \"maas-api-revoke-sub-b37291d3-ghhr9\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.397993 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.397915 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:28.678953 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:28.678924 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb004e7b4_0d12_4916_95ff_16dc7ddc407d.slice/crio-8ee4f3756b0f967973c184faaf2b4d3820afe85cc58beccac28a860723cfd65f WatchSource:0}: Error finding container 8ee4f3756b0f967973c184faaf2b4d3820afe85cc58beccac28a860723cfd65f: Status 404 returned error can't find the container with id 8ee4f3756b0f967973c184faaf2b4d3820afe85cc58beccac28a860723cfd65f Nov 01 22:35:28.682652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:28.682625 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9"] Nov 01 22:35:29.284392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.284351 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" event={"ID":"b004e7b4-0d12-4916-95ff-16dc7ddc407d","Type":"ContainerStarted","Data":"51d40b55b4d1e0072304bfd74960ddad87a7b56bfcf06107a4e24236ecc9b454"} Nov 01 22:35:29.284871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.284436 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" event={"ID":"b004e7b4-0d12-4916-95ff-16dc7ddc407d","Type":"ContainerStarted","Data":"8ee4f3756b0f967973c184faaf2b4d3820afe85cc58beccac28a860723cfd65f"} Nov 01 22:35:29.311018 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.310966 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" podStartSLOduration=1.3109523539999999 podStartE2EDuration="1.310952354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:29.304851868 +0000 UTC m=+741.953351103" watchObservedRunningTime="2026-11-01 22:35:29.310952354 +0000 UTC m=+741.959451619" Nov 01 22:35:29.463559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.463521 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:29.497964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.497932 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b26bb0-edc8-43da-9237-5ba2598af0fd-kube-api-access-hkqvr\" (UniqueName: \"kubernetes.io/projected/f8b26bb0-edc8-43da-9237-5ba2598af0fd-kube-api-access-hkqvr\") pod \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " Nov 01 22:35:29.498083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.498044 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-openshift-ca\") pod \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " Nov 01 22:35:29.498194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.498181 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-xks-ca\") pod \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\" (UID: \"f8b26bb0-edc8-43da-9237-5ba2598af0fd\") " Nov 01 22:35:29.498428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.498404 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-openshift-ca" pod "f8b26bb0-edc8-43da-9237-5ba2598af0fd" (UID: "f8b26bb0-edc8-43da-9237-5ba2598af0fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:29.498619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.498600 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:29.500070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.500048 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b26bb0-edc8-43da-9237-5ba2598af0fd-kube-api-access-hkqvr" pod "f8b26bb0-edc8-43da-9237-5ba2598af0fd" (UID: "f8b26bb0-edc8-43da-9237-5ba2598af0fd"). InnerVolumeSpecName "kube-api-access-hkqvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:29.500159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.500078 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-xks-ca" pod "f8b26bb0-edc8-43da-9237-5ba2598af0fd" (UID: "f8b26bb0-edc8-43da-9237-5ba2598af0fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:29.600069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.600040 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b26bb0-edc8-43da-9237-5ba2598af0fd-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:29.600069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:29.600068 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkqvr\" (UniqueName: \"kubernetes.io/projected/f8b26bb0-edc8-43da-9237-5ba2598af0fd-kube-api-access-hkqvr\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:30.295448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.295413 6287 generic.go:356] "Generic (PLEG): container finished" podID="b004e7b4-0d12-4916-95ff-16dc7ddc407d" containerID="51d40b55b4d1e0072304bfd74960ddad87a7b56bfcf06107a4e24236ecc9b454" exitCode=0 Nov 01 22:35:30.295911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.295473 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" event={"ID":"b004e7b4-0d12-4916-95ff-16dc7ddc407d","Type":"ContainerDied","Data":"51d40b55b4d1e0072304bfd74960ddad87a7b56bfcf06107a4e24236ecc9b454"} Nov 01 22:35:30.297056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.297033 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" Nov 01 22:35:30.297200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.297064 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv" event={"ID":"f8b26bb0-edc8-43da-9237-5ba2598af0fd","Type":"ContainerDied","Data":"53cab7b5be5d24a96c418b0e831a6f754916b1669dfd3ccb5e4c4c5102a8671b"} Nov 01 22:35:30.297200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.297127 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53cab7b5be5d24a96c418b0e831a6f754916b1669dfd3ccb5e4c4c5102a8671b" Nov 01 22:35:30.652486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.652412 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx"] Nov 01 22:35:30.653040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.653027 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f8b26bb0-edc8-43da-9237-5ba2598af0fd" containerName="revoke-keys" Nov 01 22:35:30.653078 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.653041 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b26bb0-edc8-43da-9237-5ba2598af0fd" containerName="revoke-keys" Nov 01 22:35:30.653209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.653199 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b26bb0-edc8-43da-9237-5ba2598af0fd" containerName="revoke-keys" Nov 01 22:35:30.657215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.657197 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.667403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.667376 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx"] Nov 01 22:35:30.716782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.716749 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b7bf86-trkkx\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.717175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.717152 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25vw\" (UniqueName: \"kubernetes.io/projected/dc90e4f4-887b-4862-af1c-7d3af401e15c-kube-api-access-k25vw\") pod \"maas-api-revoke-sub-f2b7bf86-trkkx\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.717317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.717194 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b7bf86-trkkx\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.818745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.818717 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25vw\" (UniqueName: \"kubernetes.io/projected/dc90e4f4-887b-4862-af1c-7d3af401e15c-kube-api-access-k25vw\") pod \"maas-api-revoke-sub-f2b7bf86-trkkx\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.818745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.818753 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b7bf86-trkkx\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.818927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.818804 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b7bf86-trkkx\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.819588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.819566 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b7bf86-trkkx\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.821246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.821227 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b7bf86-trkkx\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.826539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.826518 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25vw\" (UniqueName: \"kubernetes.io/projected/dc90e4f4-887b-4862-af1c-7d3af401e15c-kube-api-access-k25vw\") pod \"maas-api-revoke-sub-f2b7bf86-trkkx\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:30.975552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:30.975478 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:31.307060 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:31.307031 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc90e4f4_887b_4862_af1c_7d3af401e15c.slice/crio-9192ad7bfa20c3e477acfdf946d1a3b86f7ea64870b94560e3fbad3023505962 WatchSource:0}: Error finding container 9192ad7bfa20c3e477acfdf946d1a3b86f7ea64870b94560e3fbad3023505962: Status 404 returned error can't find the container with id 9192ad7bfa20c3e477acfdf946d1a3b86f7ea64870b94560e3fbad3023505962 Nov 01 22:35:31.308479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.308453 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx"] Nov 01 22:35:31.481759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.481734 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:31.530127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.530078 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b004e7b4-0d12-4916-95ff-16dc7ddc407d-kube-api-access-xwzjx\" (UniqueName: \"kubernetes.io/projected/b004e7b4-0d12-4916-95ff-16dc7ddc407d-kube-api-access-xwzjx\") pod \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " Nov 01 22:35:31.530401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.530380 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-openshift-ca\") pod \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " Nov 01 22:35:31.530492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.530425 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-xks-ca\") pod \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\" (UID: \"b004e7b4-0d12-4916-95ff-16dc7ddc407d\") " Nov 01 22:35:31.530744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.530720 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-openshift-ca" pod "b004e7b4-0d12-4916-95ff-16dc7ddc407d" (UID: "b004e7b4-0d12-4916-95ff-16dc7ddc407d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:31.531324 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.531302 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:31.532618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.532594 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b004e7b4-0d12-4916-95ff-16dc7ddc407d-kube-api-access-xwzjx" pod "b004e7b4-0d12-4916-95ff-16dc7ddc407d" (UID: "b004e7b4-0d12-4916-95ff-16dc7ddc407d"). InnerVolumeSpecName "kube-api-access-xwzjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:31.532721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.532662 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-xks-ca" pod "b004e7b4-0d12-4916-95ff-16dc7ddc407d" (UID: "b004e7b4-0d12-4916-95ff-16dc7ddc407d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:31.632396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.632373 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b004e7b4-0d12-4916-95ff-16dc7ddc407d-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:31.632396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:31.632396 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwzjx\" (UniqueName: \"kubernetes.io/projected/b004e7b4-0d12-4916-95ff-16dc7ddc407d-kube-api-access-xwzjx\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:32.172851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.172816 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2"] Nov 01 22:35:32.173792 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.173764 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b004e7b4-0d12-4916-95ff-16dc7ddc407d" containerName="revoke-keys" Nov 01 22:35:32.173792 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.173793 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b004e7b4-0d12-4916-95ff-16dc7ddc407d" containerName="revoke-keys" Nov 01 22:35:32.173984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.173972 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b004e7b4-0d12-4916-95ff-16dc7ddc407d" containerName="revoke-keys" Nov 01 22:35:32.178188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.178169 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.187167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.187142 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2"] Nov 01 22:35:32.244376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.244344 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6w5w\" (UniqueName: \"kubernetes.io/projected/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-kube-api-access-h6w5w\") pod \"maas-api-revoke-sub-dfd5e0d6-5jgw2\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.244508 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.244386 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfd5e0d6-5jgw2\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.244642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.244622 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfd5e0d6-5jgw2\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.319002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.318969 6287 generic.go:356] "Generic (PLEG): container finished" podID="dc90e4f4-887b-4862-af1c-7d3af401e15c" containerID="52b6ce2810b25aae610a8dc8174dda987f567a6e52c3057e738db7870142636a" exitCode=0 Nov 01 22:35:32.319344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.319053 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" event={"ID":"dc90e4f4-887b-4862-af1c-7d3af401e15c","Type":"ContainerDied","Data":"52b6ce2810b25aae610a8dc8174dda987f567a6e52c3057e738db7870142636a"} Nov 01 22:35:32.319344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.319104 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" event={"ID":"dc90e4f4-887b-4862-af1c-7d3af401e15c","Type":"ContainerStarted","Data":"9192ad7bfa20c3e477acfdf946d1a3b86f7ea64870b94560e3fbad3023505962"} Nov 01 22:35:32.320640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.320622 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" Nov 01 22:35:32.320758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.320662 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9" event={"ID":"b004e7b4-0d12-4916-95ff-16dc7ddc407d","Type":"ContainerDied","Data":"8ee4f3756b0f967973c184faaf2b4d3820afe85cc58beccac28a860723cfd65f"} Nov 01 22:35:32.320758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.320705 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ee4f3756b0f967973c184faaf2b4d3820afe85cc58beccac28a860723cfd65f" Nov 01 22:35:32.346233 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.346210 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfd5e0d6-5jgw2\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.346417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.346397 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6w5w\" (UniqueName: \"kubernetes.io/projected/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-kube-api-access-h6w5w\") pod \"maas-api-revoke-sub-dfd5e0d6-5jgw2\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.346484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.346440 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfd5e0d6-5jgw2\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.346909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.346887 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfd5e0d6-5jgw2\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.349031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.349010 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfd5e0d6-5jgw2\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.354397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.354377 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6w5w\" (UniqueName: \"kubernetes.io/projected/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-kube-api-access-h6w5w\") pod \"maas-api-revoke-sub-dfd5e0d6-5jgw2\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.504060 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.504019 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:32.789813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:32.789780 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2"] Nov 01 22:35:32.791848 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:32.791802 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a3b7f8_7d5e_46ca_9b1a_711ca971e0fb.slice/crio-75d8c1ec8f1e183686a2afa520ef8d981637d4be0830c9e4119c1015ffc7b9c1 WatchSource:0}: Error finding container 75d8c1ec8f1e183686a2afa520ef8d981637d4be0830c9e4119c1015ffc7b9c1: Status 404 returned error can't find the container with id 75d8c1ec8f1e183686a2afa520ef8d981637d4be0830c9e4119c1015ffc7b9c1 Nov 01 22:35:33.330243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.330204 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" event={"ID":"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb","Type":"ContainerStarted","Data":"f5c6ebb8c103dfed5f3a47e9a7d0faaebc89e7c95150a3be9804742b5aafe576"} Nov 01 22:35:33.330674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.330268 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" event={"ID":"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb","Type":"ContainerStarted","Data":"75d8c1ec8f1e183686a2afa520ef8d981637d4be0830c9e4119c1015ffc7b9c1"} Nov 01 22:35:33.352815 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.352755 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" podStartSLOduration=1.352739835 podStartE2EDuration="1.352739835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:33.348026976 +0000 UTC m=+745.996526211" watchObservedRunningTime="2026-11-01 22:35:33.352739835 +0000 UTC m=+746.001239107" Nov 01 22:35:33.516104 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.516080 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:33.567079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.567044 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-xks-ca\") pod \"dc90e4f4-887b-4862-af1c-7d3af401e15c\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " Nov 01 22:35:33.567277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.567250 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-openshift-ca\") pod \"dc90e4f4-887b-4862-af1c-7d3af401e15c\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " Nov 01 22:35:33.567447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.567416 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc90e4f4-887b-4862-af1c-7d3af401e15c-kube-api-access-k25vw\" (UniqueName: \"kubernetes.io/projected/dc90e4f4-887b-4862-af1c-7d3af401e15c-kube-api-access-k25vw\") pod \"dc90e4f4-887b-4862-af1c-7d3af401e15c\" (UID: \"dc90e4f4-887b-4862-af1c-7d3af401e15c\") " Nov 01 22:35:33.567621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.567594 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-openshift-ca" pod "dc90e4f4-887b-4862-af1c-7d3af401e15c" (UID: "dc90e4f4-887b-4862-af1c-7d3af401e15c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:33.568207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.568174 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:33.569911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.569876 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-xks-ca" pod "dc90e4f4-887b-4862-af1c-7d3af401e15c" (UID: "dc90e4f4-887b-4862-af1c-7d3af401e15c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:33.570032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.569897 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc90e4f4-887b-4862-af1c-7d3af401e15c-kube-api-access-k25vw" pod "dc90e4f4-887b-4862-af1c-7d3af401e15c" (UID: "dc90e4f4-887b-4862-af1c-7d3af401e15c"). InnerVolumeSpecName "kube-api-access-k25vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:33.669238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.669212 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25vw\" (UniqueName: \"kubernetes.io/projected/dc90e4f4-887b-4862-af1c-7d3af401e15c-kube-api-access-k25vw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:33.669238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:33.669237 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc90e4f4-887b-4862-af1c-7d3af401e15c-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:34.414134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:34.413969 6287 generic.go:356] "Generic (PLEG): container finished" podID="06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb" containerID="f5c6ebb8c103dfed5f3a47e9a7d0faaebc89e7c95150a3be9804742b5aafe576" exitCode=0 Nov 01 22:35:34.414605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:34.414173 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" event={"ID":"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb","Type":"ContainerDied","Data":"f5c6ebb8c103dfed5f3a47e9a7d0faaebc89e7c95150a3be9804742b5aafe576"} Nov 01 22:35:34.464136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:34.455566 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" event={"ID":"dc90e4f4-887b-4862-af1c-7d3af401e15c","Type":"ContainerDied","Data":"9192ad7bfa20c3e477acfdf946d1a3b86f7ea64870b94560e3fbad3023505962"} Nov 01 22:35:34.464136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:34.455637 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9192ad7bfa20c3e477acfdf946d1a3b86f7ea64870b94560e3fbad3023505962" Nov 01 22:35:34.464136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:34.455728 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx" Nov 01 22:35:35.724824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:35.723214 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:35.746146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:35.745191 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:35.769405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:35.769343 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:35.836320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:35.836215 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:35:35.868283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:35.868200 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:35.878282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:35.878197 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:35.929146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:35.926679 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.290150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.289808 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.389149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.385211 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.604000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.603898 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.771139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.737332 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.772006 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.759646 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.772791 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.772767 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:35:36.772958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.772943 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:35:36.790008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.789968 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.825881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.810327 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:35:36.858187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.844256 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:36.885136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.860163 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:35:37.010182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.885386 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.010624 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.885426 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:35:37.031524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.010985 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:35:37.033998 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.963229 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:35:37.035140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.963280 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.036172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.963301 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:35:37.041167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.963320 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:35:37.048356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.048333 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:35:37.055170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.992237 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.055370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:36.886861 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.134943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.067081 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:35:37.381365 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.381323 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:35:37.382420 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.382398 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:35:37.458193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.449787 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.483363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.483324 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.618139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.590365 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:37.665139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.620673 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.800140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.773705 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.965722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.797239 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:35:37.985138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.800677 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:37.996142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:37.859181 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.017236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.006969 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-kube-api-access-h6w5w\" (UniqueName: \"kubernetes.io/projected/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-kube-api-access-h6w5w\") pod \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " Nov 01 22:35:38.017236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.007332 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-xks-ca\") pod \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " Nov 01 22:35:38.017236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.007412 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-openshift-ca\") pod \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\" (UID: \"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb\") " Nov 01 22:35:38.017236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.010496 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.342538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.335210 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.342538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.335320 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.342538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.336945 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.342538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.338256 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.342538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.340347 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-openshift-ca" pod "06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb" (UID: "06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:38.342538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.340418 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:38.342538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.340450 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.367593 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.367553 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:38.407141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.404621 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:35:38.407141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.404895 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:35:38.407141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.405652 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.417326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.417279 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:35:38.448640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.448602 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:35:38.687132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.687080 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:35:38.687563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.687546 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:35:38.687857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.687845 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:35:38.689093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.521244 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.710543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.706961 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-kube-api-access-h6w5w" pod "06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb" (UID: "06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb"). InnerVolumeSpecName "kube-api-access-h6w5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:38.739211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.730719 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6w5w\" (UniqueName: \"kubernetes.io/projected/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-kube-api-access-h6w5w\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:38.835347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.825893 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:38.835347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.825969 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.881434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.880379 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:38.954472 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.954365 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.169143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.970185 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:35:39.182985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.970245 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:35:39.182985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:38.998662 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.182985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.098424 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:35:39.182985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.098623 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:35:39.182985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.115607 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:35:39.182985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.142621 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-xks-ca" pod "06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb" (UID: "06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:39.241818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.241779 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:39.242105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.242074 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.263357 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.263326 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:39.335531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.335489 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.361138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.337226 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.414137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.369585 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.418132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.386843 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:35:39.418132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.405403 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:35:39.422129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.377709 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded" Nov 01 22:35:39.427141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.423866 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:35:39.434511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.405352 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.464218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.459134 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.518688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.518593 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:35:39.530348 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.530319 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" Nov 01 22:35:39.693272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.693227 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:39.776391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.776300 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:35:39.776683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.776665 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:35:39.776841 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.776817 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2" event={"ID":"06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb","Type":"ContainerDied","Data":"75d8c1ec8f1e183686a2afa520ef8d981637d4be0830c9e4119c1015ffc7b9c1"} Nov 01 22:35:39.776995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.776977 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d8c1ec8f1e183686a2afa520ef8d981637d4be0830c9e4119c1015ffc7b9c1" Nov 01 22:35:39.806521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.806497 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:35:39.900342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.900163 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:35:39.908283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.908248 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:35:39.927391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.927366 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:35:39.948304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.948253 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:35:39.955153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.949848 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:35:39.955153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.950428 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:35:39.956985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:39.956963 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:35:43.158616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.158578 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t"] Nov 01 22:35:43.159264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.159249 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dc90e4f4-887b-4862-af1c-7d3af401e15c" containerName="revoke-keys" Nov 01 22:35:43.159319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.159265 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc90e4f4-887b-4862-af1c-7d3af401e15c" containerName="revoke-keys" Nov 01 22:35:43.159319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.159273 6287 cpu_manager.go:414] "containerMap: removing container" podUID="06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb" containerName="revoke-keys" Nov 01 22:35:43.159319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.159278 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb" containerName="revoke-keys" Nov 01 22:35:43.159412 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.159395 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc90e4f4-887b-4862-af1c-7d3af401e15c" containerName="revoke-keys" Nov 01 22:35:43.159412 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.159405 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb" containerName="revoke-keys" Nov 01 22:35:43.163039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.163019 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.167749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.167723 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:35:43.167749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.167731 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:43.167953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.167731 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:35:43.167953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.167781 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:35:43.170097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.170074 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t"] Nov 01 22:35:43.232134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.232080 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be4916a-lcf6t\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.232360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.232337 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be4916a-lcf6t\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.232434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.232402 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xqv\" (UniqueName: \"kubernetes.io/projected/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-kube-api-access-d4xqv\") pod \"maas-api-revoke-sub-6be4916a-lcf6t\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.334462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.334423 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be4916a-lcf6t\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.334717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.334698 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be4916a-lcf6t\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.334791 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.334777 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xqv\" (UniqueName: \"kubernetes.io/projected/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-kube-api-access-d4xqv\") pod \"maas-api-revoke-sub-6be4916a-lcf6t\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.335344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.335319 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be4916a-lcf6t\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.336973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.336945 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be4916a-lcf6t\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.342953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.342930 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xqv\" (UniqueName: \"kubernetes.io/projected/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-kube-api-access-d4xqv\") pod \"maas-api-revoke-sub-6be4916a-lcf6t\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.480009 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.479931 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:43.524575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.524535 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg"] Nov 01 22:35:43.535154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.533350 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.542633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.542412 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg"] Nov 01 22:35:43.639553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.639504 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-014bf358-dx9cg\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.639896 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.639876 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t76fn\" (UniqueName: \"kubernetes.io/projected/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-kube-api-access-t76fn\") pod \"maas-api-revoke-sub-014bf358-dx9cg\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.640163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.640142 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-014bf358-dx9cg\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.744351 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.744319 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-014bf358-dx9cg\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.744550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.744372 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t76fn\" (UniqueName: \"kubernetes.io/projected/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-kube-api-access-t76fn\") pod \"maas-api-revoke-sub-014bf358-dx9cg\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.744550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.744449 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-014bf358-dx9cg\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.745103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.745080 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-014bf358-dx9cg\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.746908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.746890 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-014bf358-dx9cg\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.752102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.752080 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t76fn\" (UniqueName: \"kubernetes.io/projected/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-kube-api-access-t76fn\") pod \"maas-api-revoke-sub-014bf358-dx9cg\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:43.772644 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:43.772619 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2415d96_c6cc_4d3d_a89d_bcdb4b356ee9.slice/crio-4fea27285d3801c5dbf208fcf8a38ee5d82969d66d49d133469f34193181bd53 WatchSource:0}: Error finding container 4fea27285d3801c5dbf208fcf8a38ee5d82969d66d49d133469f34193181bd53: Status 404 returned error can't find the container with id 4fea27285d3801c5dbf208fcf8a38ee5d82969d66d49d133469f34193181bd53 Nov 01 22:35:43.773995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.773973 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t"] Nov 01 22:35:43.855264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:43.855240 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:44.136270 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:44.136232 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4f0e3b_02b8_47ed_85ec_d93b07ba9bcc.slice/crio-faac0c8b58a0573185ab70c85761c6ed480afdd71d1dd6b115b0095f63e38efa WatchSource:0}: Error finding container faac0c8b58a0573185ab70c85761c6ed480afdd71d1dd6b115b0095f63e38efa: Status 404 returned error can't find the container with id faac0c8b58a0573185ab70c85761c6ed480afdd71d1dd6b115b0095f63e38efa Nov 01 22:35:44.137997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:44.137870 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg"] Nov 01 22:35:44.592759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:44.592717 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" event={"ID":"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9","Type":"ContainerStarted","Data":"1f25f97376d8220221c602d3f0c5311319542a33678cc0835134d211bd7a8f46"} Nov 01 22:35:44.593419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:44.593383 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" event={"ID":"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9","Type":"ContainerStarted","Data":"4fea27285d3801c5dbf208fcf8a38ee5d82969d66d49d133469f34193181bd53"} Nov 01 22:35:44.594954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:44.594923 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" event={"ID":"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc","Type":"ContainerStarted","Data":"d8d33ec10b55b63c5949cdb7c898a1acadfdd60c5a6130f2201121a3371446d6"} Nov 01 22:35:44.595080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:44.594979 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" event={"ID":"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc","Type":"ContainerStarted","Data":"faac0c8b58a0573185ab70c85761c6ed480afdd71d1dd6b115b0095f63e38efa"} Nov 01 22:35:44.614487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:44.614445 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" podStartSLOduration=1.6144302910000001 podStartE2EDuration="1.614430291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:44.611104969 +0000 UTC m=+757.259604196" watchObservedRunningTime="2026-11-01 22:35:44.614430291 +0000 UTC m=+757.262929529" Nov 01 22:35:44.642800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:44.642750 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" podStartSLOduration=1.642734211 podStartE2EDuration="1.642734211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:44.633421584 +0000 UTC m=+757.281920818" watchObservedRunningTime="2026-11-01 22:35:44.642734211 +0000 UTC m=+757.291233446" Nov 01 22:35:45.590975 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:35:45.590931 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:35:45.605175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:45.605141 6287 generic.go:356] "Generic (PLEG): container finished" podID="f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9" containerID="1f25f97376d8220221c602d3f0c5311319542a33678cc0835134d211bd7a8f46" exitCode=0 Nov 01 22:35:45.605562 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:45.605177 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" event={"ID":"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9","Type":"ContainerDied","Data":"1f25f97376d8220221c602d3f0c5311319542a33678cc0835134d211bd7a8f46"} Nov 01 22:35:45.606909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:45.606885 6287 generic.go:356] "Generic (PLEG): container finished" podID="0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc" containerID="d8d33ec10b55b63c5949cdb7c898a1acadfdd60c5a6130f2201121a3371446d6" exitCode=0 Nov 01 22:35:45.607073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:45.606967 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" event={"ID":"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc","Type":"ContainerDied","Data":"d8d33ec10b55b63c5949cdb7c898a1acadfdd60c5a6130f2201121a3371446d6"} Nov 01 22:35:46.892166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:46.892138 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:46.894582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:46.894559 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:47.005506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.005472 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-openshift-ca\") pod \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " Nov 01 22:35:47.005684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.005568 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-kube-api-access-t76fn\" (UniqueName: \"kubernetes.io/projected/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-kube-api-access-t76fn\") pod \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " Nov 01 22:35:47.005684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.005652 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-xks-ca\") pod \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " Nov 01 22:35:47.005796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.005727 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-openshift-ca\") pod \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\" (UID: \"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc\") " Nov 01 22:35:47.005846 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.005818 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-xks-ca\") pod \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " Nov 01 22:35:47.005900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.005860 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-openshift-ca" pod "f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9" (UID: "f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:47.005991 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.005970 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-kube-api-access-d4xqv\" (UniqueName: \"kubernetes.io/projected/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-kube-api-access-d4xqv\") pod \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\" (UID: \"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9\") " Nov 01 22:35:47.006148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.006103 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-openshift-ca" pod "0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc" (UID: "0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:47.006776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.006748 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:47.006895 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.006781 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:47.009151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.008002 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-xks-ca" pod "f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9" (UID: "f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:47.009151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.008071 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-xks-ca" pod "0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc" (UID: "0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:47.009151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.008184 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-kube-api-access-t76fn" pod "0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc" (UID: "0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc"). InnerVolumeSpecName "kube-api-access-t76fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:47.009151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.008301 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-kube-api-access-d4xqv" pod "f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9" (UID: "f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9"). InnerVolumeSpecName "kube-api-access-d4xqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:47.108243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.108218 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t76fn\" (UniqueName: \"kubernetes.io/projected/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-kube-api-access-t76fn\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:47.108243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.108240 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:47.108243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.108250 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:47.108438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.108260 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xqv\" (UniqueName: \"kubernetes.io/projected/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9-kube-api-access-d4xqv\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:47.626036 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.626009 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" Nov 01 22:35:47.626204 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.626034 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t" event={"ID":"f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9","Type":"ContainerDied","Data":"4fea27285d3801c5dbf208fcf8a38ee5d82969d66d49d133469f34193181bd53"} Nov 01 22:35:47.626204 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.626085 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fea27285d3801c5dbf208fcf8a38ee5d82969d66d49d133469f34193181bd53" Nov 01 22:35:47.627527 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.627502 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" event={"ID":"0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc","Type":"ContainerDied","Data":"faac0c8b58a0573185ab70c85761c6ed480afdd71d1dd6b115b0095f63e38efa"} Nov 01 22:35:47.627640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.627528 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg" Nov 01 22:35:47.627640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:47.627545 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faac0c8b58a0573185ab70c85761c6ed480afdd71d1dd6b115b0095f63e38efa" Nov 01 22:35:48.178929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.178898 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z"] Nov 01 22:35:48.179558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.179544 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc" containerName="revoke-keys" Nov 01 22:35:48.179603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.179559 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc" containerName="revoke-keys" Nov 01 22:35:48.179603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.179580 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9" containerName="revoke-keys" Nov 01 22:35:48.179603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.179585 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9" containerName="revoke-keys" Nov 01 22:35:48.179789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.179687 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc" containerName="revoke-keys" Nov 01 22:35:48.179789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.179698 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9" containerName="revoke-keys" Nov 01 22:35:48.184851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.184827 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.198005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.191322 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:35:48.198005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.191498 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:35:48.198005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.191547 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:48.198005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.192081 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:35:48.198005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.192332 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z"] Nov 01 22:35:48.329786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.329752 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-158c7ee6-d8n7z\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.329952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.329884 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jr52\" (UniqueName: \"kubernetes.io/projected/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-kube-api-access-2jr52\") pod \"maas-api-revoke-sub-158c7ee6-d8n7z\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.329952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.329922 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-158c7ee6-d8n7z\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.431140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.431044 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-158c7ee6-d8n7z\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.431286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.431174 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jr52\" (UniqueName: \"kubernetes.io/projected/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-kube-api-access-2jr52\") pod \"maas-api-revoke-sub-158c7ee6-d8n7z\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.431286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.431217 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-158c7ee6-d8n7z\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.431909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.431883 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-158c7ee6-d8n7z\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.433483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.433458 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-158c7ee6-d8n7z\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.439270 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.439247 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jr52\" (UniqueName: \"kubernetes.io/projected/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-kube-api-access-2jr52\") pod \"maas-api-revoke-sub-158c7ee6-d8n7z\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.512183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.512155 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:48.792219 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:48.792187 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85ea45c_e2dd_4119_a0f1_1b78c40bb867.slice/crio-55e2f51ac40ef03439dd60f34fa170642d073844aef3e03c12b350e519ed7742 WatchSource:0}: Error finding container 55e2f51ac40ef03439dd60f34fa170642d073844aef3e03c12b350e519ed7742: Status 404 returned error can't find the container with id 55e2f51ac40ef03439dd60f34fa170642d073844aef3e03c12b350e519ed7742 Nov 01 22:35:48.796971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:48.796948 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z"] Nov 01 22:35:49.664125 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:49.664088 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" event={"ID":"a85ea45c-e2dd-4119-a0f1-1b78c40bb867","Type":"ContainerStarted","Data":"a32336e110ecd0e514ff70331694699c44fae5e1e5a6af1eab64386e544e8374"} Nov 01 22:35:49.664577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:49.664175 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" event={"ID":"a85ea45c-e2dd-4119-a0f1-1b78c40bb867","Type":"ContainerStarted","Data":"55e2f51ac40ef03439dd60f34fa170642d073844aef3e03c12b350e519ed7742"} Nov 01 22:35:49.726056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:49.726012 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" podStartSLOduration=1.725998734 podStartE2EDuration="1.725998734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:49.721622329 +0000 UTC m=+762.370121565" watchObservedRunningTime="2026-11-01 22:35:49.725998734 +0000 UTC m=+762.374497968" Nov 01 22:35:50.674284 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:50.674242 6287 generic.go:356] "Generic (PLEG): container finished" podID="a85ea45c-e2dd-4119-a0f1-1b78c40bb867" containerID="a32336e110ecd0e514ff70331694699c44fae5e1e5a6af1eab64386e544e8374" exitCode=0 Nov 01 22:35:50.674682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:50.674330 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" event={"ID":"a85ea45c-e2dd-4119-a0f1-1b78c40bb867","Type":"ContainerDied","Data":"a32336e110ecd0e514ff70331694699c44fae5e1e5a6af1eab64386e544e8374"} Nov 01 22:35:51.843937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:51.843910 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:51.984771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:51.984673 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-xks-ca\") pod \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " Nov 01 22:35:51.984956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:51.984775 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-kube-api-access-2jr52\" (UniqueName: \"kubernetes.io/projected/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-kube-api-access-2jr52\") pod \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " Nov 01 22:35:51.985146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:51.985107 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-openshift-ca\") pod \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\" (UID: \"a85ea45c-e2dd-4119-a0f1-1b78c40bb867\") " Nov 01 22:35:51.985412 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:51.985391 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-openshift-ca" pod "a85ea45c-e2dd-4119-a0f1-1b78c40bb867" (UID: "a85ea45c-e2dd-4119-a0f1-1b78c40bb867"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:51.985816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:51.985797 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:51.987392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:51.987364 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-xks-ca" pod "a85ea45c-e2dd-4119-a0f1-1b78c40bb867" (UID: "a85ea45c-e2dd-4119-a0f1-1b78c40bb867"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:51.988637 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:51.988612 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-kube-api-access-2jr52" pod "a85ea45c-e2dd-4119-a0f1-1b78c40bb867" (UID: "a85ea45c-e2dd-4119-a0f1-1b78c40bb867"). InnerVolumeSpecName "kube-api-access-2jr52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:52.087268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:52.087240 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:52.087268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:52.087266 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jr52\" (UniqueName: \"kubernetes.io/projected/a85ea45c-e2dd-4119-a0f1-1b78c40bb867-kube-api-access-2jr52\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:52.731426 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:52.731395 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" Nov 01 22:35:52.731844 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:52.731455 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z" event={"ID":"a85ea45c-e2dd-4119-a0f1-1b78c40bb867","Type":"ContainerDied","Data":"55e2f51ac40ef03439dd60f34fa170642d073844aef3e03c12b350e519ed7742"} Nov 01 22:35:52.733203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:52.733178 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55e2f51ac40ef03439dd60f34fa170642d073844aef3e03c12b350e519ed7742" Nov 01 22:35:53.494906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.494865 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk"] Nov 01 22:35:53.495496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.495479 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a85ea45c-e2dd-4119-a0f1-1b78c40bb867" containerName="revoke-keys" Nov 01 22:35:53.495496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.495494 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85ea45c-e2dd-4119-a0f1-1b78c40bb867" containerName="revoke-keys" Nov 01 22:35:53.495632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.495603 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85ea45c-e2dd-4119-a0f1-1b78c40bb867" containerName="revoke-keys" Nov 01 22:35:53.499820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.499796 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk"] Nov 01 22:35:53.499945 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.499884 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.506505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.506480 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:35:53.506656 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.506553 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:53.506656 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.506556 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:35:53.506656 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.506563 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:35:53.524932 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.524855 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w277\" (UniqueName: \"kubernetes.io/projected/dc5be362-d503-4e52-9896-844827e5f7f0-kube-api-access-9w277\") pod \"maas-api-revoke-sub-16bec70b-7x7mk\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.525423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.525401 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16bec70b-7x7mk\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.525610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.525513 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16bec70b-7x7mk\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.627571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.627530 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w277\" (UniqueName: \"kubernetes.io/projected/dc5be362-d503-4e52-9896-844827e5f7f0-kube-api-access-9w277\") pod \"maas-api-revoke-sub-16bec70b-7x7mk\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.627729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.627641 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16bec70b-7x7mk\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.627729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.627686 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16bec70b-7x7mk\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.628325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.628294 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16bec70b-7x7mk\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.630071 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.630052 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16bec70b-7x7mk\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.638305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.638281 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w277\" (UniqueName: \"kubernetes.io/projected/dc5be362-d503-4e52-9896-844827e5f7f0-kube-api-access-9w277\") pod \"maas-api-revoke-sub-16bec70b-7x7mk\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.693284 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.693249 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx"] Nov 01 22:35:53.698121 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.698086 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:53.708786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.708755 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx"] Nov 01 22:35:53.728520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.728490 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdws\" (UniqueName: \"kubernetes.io/projected/555649a3-ccf4-48c6-95e7-ee7623f6ba84-kube-api-access-rtdws\") pod \"maas-api-revoke-sub-58ef0be5-z4cpx\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:53.728657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.728536 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58ef0be5-z4cpx\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:53.728657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.728623 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58ef0be5-z4cpx\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:53.826178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.826069 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:53.829891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.829869 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdws\" (UniqueName: \"kubernetes.io/projected/555649a3-ccf4-48c6-95e7-ee7623f6ba84-kube-api-access-rtdws\") pod \"maas-api-revoke-sub-58ef0be5-z4cpx\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:53.830032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.829914 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58ef0be5-z4cpx\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:53.830032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.829952 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58ef0be5-z4cpx\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:53.830853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.830826 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58ef0be5-z4cpx\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:53.832470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.832452 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58ef0be5-z4cpx\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:53.838242 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:53.838224 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdws\" (UniqueName: \"kubernetes.io/projected/555649a3-ccf4-48c6-95e7-ee7623f6ba84-kube-api-access-rtdws\") pod \"maas-api-revoke-sub-58ef0be5-z4cpx\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:54.017522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:54.017487 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:54.121902 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:54.121868 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5be362_d503_4e52_9896_844827e5f7f0.slice/crio-d9b5377f49de84de2266bfbcd7d7cbd3458385d6c5183b42050ecb7f6aed3006 WatchSource:0}: Error finding container d9b5377f49de84de2266bfbcd7d7cbd3458385d6c5183b42050ecb7f6aed3006: Status 404 returned error can't find the container with id d9b5377f49de84de2266bfbcd7d7cbd3458385d6c5183b42050ecb7f6aed3006 Nov 01 22:35:54.124018 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:54.123990 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk"] Nov 01 22:35:54.309962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:54.309925 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx"] Nov 01 22:35:54.750589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:54.750547 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" event={"ID":"dc5be362-d503-4e52-9896-844827e5f7f0","Type":"ContainerStarted","Data":"2ec4117630bd23152dd1233eaaacd4fdb88befbc5e7964bb67df0fc63dac69e9"} Nov 01 22:35:54.750997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:54.750616 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" event={"ID":"dc5be362-d503-4e52-9896-844827e5f7f0","Type":"ContainerStarted","Data":"d9b5377f49de84de2266bfbcd7d7cbd3458385d6c5183b42050ecb7f6aed3006"} Nov 01 22:35:54.752460 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:54.752435 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" event={"ID":"555649a3-ccf4-48c6-95e7-ee7623f6ba84","Type":"ContainerStarted","Data":"fdbd5d7d9a98d5e7ab16b4c7e19f7240823eb82b7d3cbd19b33229318f89265a"} Nov 01 22:35:54.752575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:54.752488 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" event={"ID":"555649a3-ccf4-48c6-95e7-ee7623f6ba84","Type":"ContainerStarted","Data":"60c21740305443d6174861b8242c3afc2278df10bb667414d83eec70476175d2"} Nov 01 22:35:54.770902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:54.770850 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" podStartSLOduration=1.7708377739999999 podStartE2EDuration="1.770837774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:54.768194415 +0000 UTC m=+767.416693650" watchObservedRunningTime="2026-11-01 22:35:54.770837774 +0000 UTC m=+767.419337030" Nov 01 22:35:54.787294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:54.787244 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" podStartSLOduration=1.7872308609999998 podStartE2EDuration="1.787230861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:54.78488472 +0000 UTC m=+767.433383957" watchObservedRunningTime="2026-11-01 22:35:54.787230861 +0000 UTC m=+767.435730127" Nov 01 22:35:55.767382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:55.767343 6287 generic.go:356] "Generic (PLEG): container finished" podID="dc5be362-d503-4e52-9896-844827e5f7f0" containerID="2ec4117630bd23152dd1233eaaacd4fdb88befbc5e7964bb67df0fc63dac69e9" exitCode=0 Nov 01 22:35:55.767926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:55.767446 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" event={"ID":"dc5be362-d503-4e52-9896-844827e5f7f0","Type":"ContainerDied","Data":"2ec4117630bd23152dd1233eaaacd4fdb88befbc5e7964bb67df0fc63dac69e9"} Nov 01 22:35:55.770029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:55.769999 6287 generic.go:356] "Generic (PLEG): container finished" podID="555649a3-ccf4-48c6-95e7-ee7623f6ba84" containerID="fdbd5d7d9a98d5e7ab16b4c7e19f7240823eb82b7d3cbd19b33229318f89265a" exitCode=0 Nov 01 22:35:55.770198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:55.770058 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" event={"ID":"555649a3-ccf4-48c6-95e7-ee7623f6ba84","Type":"ContainerDied","Data":"fdbd5d7d9a98d5e7ab16b4c7e19f7240823eb82b7d3cbd19b33229318f89265a"} Nov 01 22:35:56.979816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:56.979786 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:57.024521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.024490 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:57.129641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.129555 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-xks-ca\") pod \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " Nov 01 22:35:57.129641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.129608 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-openshift-ca\") pod \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " Nov 01 22:35:57.129864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.129745 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555649a3-ccf4-48c6-95e7-ee7623f6ba84-kube-api-access-rtdws\" (UniqueName: \"kubernetes.io/projected/555649a3-ccf4-48c6-95e7-ee7623f6ba84-kube-api-access-rtdws\") pod \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\" (UID: \"555649a3-ccf4-48c6-95e7-ee7623f6ba84\") " Nov 01 22:35:57.129864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.129797 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-xks-ca\") pod \"dc5be362-d503-4e52-9896-844827e5f7f0\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " Nov 01 22:35:57.129962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.129866 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5be362-d503-4e52-9896-844827e5f7f0-kube-api-access-9w277\" (UniqueName: \"kubernetes.io/projected/dc5be362-d503-4e52-9896-844827e5f7f0-kube-api-access-9w277\") pod \"dc5be362-d503-4e52-9896-844827e5f7f0\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " Nov 01 22:35:57.129962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.129922 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-openshift-ca" pod "555649a3-ccf4-48c6-95e7-ee7623f6ba84" (UID: "555649a3-ccf4-48c6-95e7-ee7623f6ba84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:57.130059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.130045 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-openshift-ca\") pod \"dc5be362-d503-4e52-9896-844827e5f7f0\" (UID: \"dc5be362-d503-4e52-9896-844827e5f7f0\") " Nov 01 22:35:57.130445 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.130415 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-openshift-ca" pod "dc5be362-d503-4e52-9896-844827e5f7f0" (UID: "dc5be362-d503-4e52-9896-844827e5f7f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:35:57.130785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.130755 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:57.130785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.130775 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:57.131892 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.131868 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-xks-ca" pod "555649a3-ccf4-48c6-95e7-ee7623f6ba84" (UID: "555649a3-ccf4-48c6-95e7-ee7623f6ba84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:57.132063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.132045 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-xks-ca" pod "dc5be362-d503-4e52-9896-844827e5f7f0" (UID: "dc5be362-d503-4e52-9896-844827e5f7f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:35:57.132456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.132439 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5be362-d503-4e52-9896-844827e5f7f0-kube-api-access-9w277" pod "dc5be362-d503-4e52-9896-844827e5f7f0" (UID: "dc5be362-d503-4e52-9896-844827e5f7f0"). InnerVolumeSpecName "kube-api-access-9w277". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:57.132524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.132461 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555649a3-ccf4-48c6-95e7-ee7623f6ba84-kube-api-access-rtdws" pod "555649a3-ccf4-48c6-95e7-ee7623f6ba84" (UID: "555649a3-ccf4-48c6-95e7-ee7623f6ba84"). InnerVolumeSpecName "kube-api-access-rtdws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:35:57.232671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.232638 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555649a3-ccf4-48c6-95e7-ee7623f6ba84-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:57.232671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.232670 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtdws\" (UniqueName: \"kubernetes.io/projected/555649a3-ccf4-48c6-95e7-ee7623f6ba84-kube-api-access-rtdws\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:57.232671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.232684 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc5be362-d503-4e52-9896-844827e5f7f0-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:57.232916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.232697 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w277\" (UniqueName: \"kubernetes.io/projected/dc5be362-d503-4e52-9896-844827e5f7f0-kube-api-access-9w277\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:35:57.833421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.833318 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" event={"ID":"dc5be362-d503-4e52-9896-844827e5f7f0","Type":"ContainerDied","Data":"d9b5377f49de84de2266bfbcd7d7cbd3458385d6c5183b42050ecb7f6aed3006"} Nov 01 22:35:57.839200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.839167 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9b5377f49de84de2266bfbcd7d7cbd3458385d6c5183b42050ecb7f6aed3006" Nov 01 22:35:57.839648 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.839626 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk" Nov 01 22:35:57.847436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.847362 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" event={"ID":"555649a3-ccf4-48c6-95e7-ee7623f6ba84","Type":"ContainerDied","Data":"60c21740305443d6174861b8242c3afc2278df10bb667414d83eec70476175d2"} Nov 01 22:35:57.847595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.847456 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60c21740305443d6174861b8242c3afc2278df10bb667414d83eec70476175d2" Nov 01 22:35:57.847595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:57.847554 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx" Nov 01 22:35:58.548626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.548590 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph"] Nov 01 22:35:58.549258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.549243 6287 cpu_manager.go:414] "containerMap: removing container" podUID="555649a3-ccf4-48c6-95e7-ee7623f6ba84" containerName="revoke-keys" Nov 01 22:35:58.549300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.549260 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555649a3-ccf4-48c6-95e7-ee7623f6ba84" containerName="revoke-keys" Nov 01 22:35:58.549300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.549295 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dc5be362-d503-4e52-9896-844827e5f7f0" containerName="revoke-keys" Nov 01 22:35:58.549378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.549301 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5be362-d503-4e52-9896-844827e5f7f0" containerName="revoke-keys" Nov 01 22:35:58.549421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.549411 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5be362-d503-4e52-9896-844827e5f7f0" containerName="revoke-keys" Nov 01 22:35:58.549457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.549429 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="555649a3-ccf4-48c6-95e7-ee7623f6ba84" containerName="revoke-keys" Nov 01 22:35:58.553877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.553855 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.562942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.562910 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph"] Nov 01 22:35:58.565774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.564651 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:35:58.565774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.564764 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:35:58.565774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.565639 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:35:58.565774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.565721 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:35:58.655050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.655017 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958aa513-62rph\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.655247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.655068 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958aa513-62rph\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.655342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.655240 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc572\" (UniqueName: \"kubernetes.io/projected/45e4b2bc-b479-44cd-9532-f3ef6ae75452-kube-api-access-lc572\") pod \"maas-api-revoke-sub-958aa513-62rph\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.756591 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.756555 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958aa513-62rph\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.756737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.756617 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958aa513-62rph\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.756737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.756704 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc572\" (UniqueName: \"kubernetes.io/projected/45e4b2bc-b479-44cd-9532-f3ef6ae75452-kube-api-access-lc572\") pod \"maas-api-revoke-sub-958aa513-62rph\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.757349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.757323 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-958aa513-62rph\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.759245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.759218 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-958aa513-62rph\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.767295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.767270 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc572\" (UniqueName: \"kubernetes.io/projected/45e4b2bc-b479-44cd-9532-f3ef6ae75452-kube-api-access-lc572\") pod \"maas-api-revoke-sub-958aa513-62rph\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:58.879646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:58.879566 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:35:59.175695 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:35:59.175665 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice/crio-80567058ca7bcd3af4291f4645a3d28394f46b826d5d8fd7d7a56780f4aa9eb3 WatchSource:0}: Error finding container 80567058ca7bcd3af4291f4645a3d28394f46b826d5d8fd7d7a56780f4aa9eb3: Status 404 returned error can't find the container with id 80567058ca7bcd3af4291f4645a3d28394f46b826d5d8fd7d7a56780f4aa9eb3 Nov 01 22:35:59.177489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.177465 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph"] Nov 01 22:35:59.783627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.783587 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs"] Nov 01 22:35:59.788460 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.788432 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:35:59.815568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.815538 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs"] Nov 01 22:35:59.831501 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.831475 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz"] Nov 01 22:35:59.837592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.837489 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.863405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.862931 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz"] Nov 01 22:35:59.874476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.874443 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" event={"ID":"45e4b2bc-b479-44cd-9532-f3ef6ae75452","Type":"ContainerStarted","Data":"9a6139b9ba598472c47e600be51ae75f22ace42b0bb8bdd21fe22e1388eb4ed5"} Nov 01 22:35:59.874717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.874691 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" event={"ID":"45e4b2bc-b479-44cd-9532-f3ef6ae75452","Type":"ContainerStarted","Data":"80567058ca7bcd3af4291f4645a3d28394f46b826d5d8fd7d7a56780f4aa9eb3"} Nov 01 22:35:59.895592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.895539 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" podStartSLOduration=1.895526257 podStartE2EDuration="1.895526257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:59.892929231 +0000 UTC m=+772.541428466" watchObservedRunningTime="2026-11-01 22:35:59.895526257 +0000 UTC m=+772.544025492" Nov 01 22:35:59.974703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.974672 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.974906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.974710 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.974906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.974749 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.975056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.974921 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:35:59.975056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975010 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac045d0-c9da-4ca2-b969-804bba24373b-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:35:59.975144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975078 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:35:59.975189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975171 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:35:59.975359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975339 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.975396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975381 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.975537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975520 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:35:59.975572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975556 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.975629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975574 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9j5\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-kube-api-access-nc9j5\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.975700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975684 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.975758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975719 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:35:59.975814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975779 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfxm6\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-kube-api-access-zfxm6\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:35:59.975868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975813 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:35:59.975868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975861 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:35:59.975937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:35:59.975907 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac045d0-c9da-4ca2-b969-804bba24373b-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.076891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.076854 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.076891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.076897 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.077140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.076943 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.076975 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac045d0-c9da-4ca2-b969-804bba24373b-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.076997 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077143 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077199 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.077313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077237 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.077473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077368 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077397 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077409 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.077473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077450 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9j5\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-kube-api-access-nc9j5\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.077673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077511 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.077673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077541 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.077673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077582 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfxm6\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-kube-api-access-zfxm6\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077616 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077663 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077702 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac045d0-c9da-4ca2-b969-804bba24373b-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.077916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077737 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.077916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077367 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.077916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.077884 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac045d0-c9da-4ca2-b969-804bba24373b-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.078140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.078063 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-certs\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.078226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.078185 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.078586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.078331 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.078586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.078463 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.078775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.078595 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-credential-socket\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.078775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.078760 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-data\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.078887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.078818 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istiod-ca-cert\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.079921 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.079896 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.080066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.080037 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-envoy\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.080222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.080202 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.080779 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.080757 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac045d0-c9da-4ca2-b969-804bba24373b-istio-podinfo\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.086159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.086130 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.086269 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.086242 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9j5\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-kube-api-access-nc9j5\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.086649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.086623 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfxm6\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-kube-api-access-zfxm6\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.088292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.088251 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-istio-token\") pod \"e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.107397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.107374 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:00.161345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.161313 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:00.407499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.407461 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs"] Nov 01 22:36:00.438197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.438152 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:00.438341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.438253 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:00.438341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.438304 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:00.555426 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.555375 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz"] Nov 01 22:36:00.579676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.579539 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:00.579951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.579919 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:00.580094 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:00.580075 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:01.036462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.036424 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" event={"ID":"aac045d0-c9da-4ca2-b969-804bba24373b","Type":"ContainerStarted","Data":"6d0eb627cee83050fae9b7bbd4f40d8d1e64245d18d737ae22fe99a7242edbc3"} Nov 01 22:36:01.037038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.037012 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" event={"ID":"aac045d0-c9da-4ca2-b969-804bba24373b","Type":"ContainerStarted","Data":"6fdd85703bfab480b373751cd442848e010078902270b036cd0bf56e55874d17"} Nov 01 22:36:01.125245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.125216 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:01.232137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.216516 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:01.232137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.216560 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" event={"ID":"b5dc5043-94c8-4546-8c7b-a9c14540f45c","Type":"ContainerStarted","Data":"3c686ccbc58dfd3b8af485b6c6484bf3a88152ae37de7c71d8bd0cee76703bc6"} Nov 01 22:36:01.232137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.216590 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" event={"ID":"b5dc5043-94c8-4546-8c7b-a9c14540f45c","Type":"ContainerStarted","Data":"fd17babfb90ff0cc1ec6fde080f99bc6e974f83290070f08a4f1c6a12a0f2b2d"} Nov 01 22:36:01.232137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.151451 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:01.232137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.216658 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:01.232137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.153984 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podStartSLOduration=2.1539648160000002 podStartE2EDuration="2.153964816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:01.126807218 +0000 UTC m=+773.775306445" watchObservedRunningTime="2026-11-01 22:36:01.153964816 +0000 UTC m=+773.802464054" Nov 01 22:36:01.234710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.234674 6287 generic.go:356] "Generic (PLEG): container finished" podID="45e4b2bc-b479-44cd-9532-f3ef6ae75452" containerID="9a6139b9ba598472c47e600be51ae75f22ace42b0bb8bdd21fe22e1388eb4ed5" exitCode=0 Nov 01 22:36:01.234979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.234959 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" event={"ID":"45e4b2bc-b479-44cd-9532-f3ef6ae75452","Type":"ContainerDied","Data":"9a6139b9ba598472c47e600be51ae75f22ace42b0bb8bdd21fe22e1388eb4ed5"} Nov 01 22:36:01.244800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.206204 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:01.251647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.245005 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:01.309208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.296000 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podStartSLOduration=2.295980083 podStartE2EDuration="2.295980083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:01.26284539 +0000 UTC m=+773.911344616" watchObservedRunningTime="2026-11-01 22:36:01.295980083 +0000 UTC m=+773.944479320" Nov 01 22:36:01.913152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.863347 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:36:01.982145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.975235 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:01.982145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.975661 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:36:01.997811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.997766 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:01.999302 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:01.999268 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:02.108929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.108882 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:02.109555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.109516 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:02.175197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.162654 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:02.175197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.162719 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:02.215260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.215211 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:02.503962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.462351 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:02.503962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.462742 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:36:02.503962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.463038 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:36:02.912408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.912303 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:02.913456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.913429 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:36:02.952631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.952581 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:02.993249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.993197 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:02.993871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.993836 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:03.011941 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:02.995072 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:03.042813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.012217 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:03.099473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.013435 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:03.122810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.013467 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:03.134912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.043366 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:36:03.650174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.638005 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:36:03.650174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.638088 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:36:03.678180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.177252 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:03.678180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.329933 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:03.678180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.673883 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:03.678180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.470587 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:03.678180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.198397 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:03.678180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.239475 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:03.704464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.470553 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:03.704464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.375258 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:03.704464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.686581 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:03.704464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.470608 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:36:03.964197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.919264 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:36:03.968139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.965041 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:36:04.020498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.997303 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:04.025156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:03.997359 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:04.047198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.043270 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:36:04.124136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.092669 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:04.349737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.144681 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:04.422156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.144719 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:04.422156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.227228 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:04.422156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.415816 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:04.422156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.377250 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:04.422156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.241748 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:04.422156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.416129 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:04.500197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.476633 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:36:04.500197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.476699 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:36:04.500197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.476735 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:36:04.500197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.476767 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:36:04.500197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.476799 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:36:04.500197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.476829 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:36:04.500197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.476862 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:36:04.500197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.476890 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:36:04.577047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.576992 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Nov 01 22:36:04.660144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.652202 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:04.763941 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.725722 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:36:04.764933 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.725754 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:04.792287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.792251 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:36:04.811167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.725791 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:04.872139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.725834 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:04.872812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.725853 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:36:04.873301 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.763042 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:04.873457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.725812 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:36:04.877260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.877240 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:36:04.970213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.942737 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" event={"ID":"45e4b2bc-b479-44cd-9532-f3ef6ae75452","Type":"ContainerDied","Data":"80567058ca7bcd3af4291f4645a3d28394f46b826d5d8fd7d7a56780f4aa9eb3"} Nov 01 22:36:04.992697 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.992654 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80567058ca7bcd3af4291f4645a3d28394f46b826d5d8fd7d7a56780f4aa9eb3" Nov 01 22:36:04.993399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:04.993373 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:36:05.030302 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.030195 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:05.229268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.229215 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:36:05.234138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.230151 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:05.253163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.248380 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:05.253468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.253428 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:05.256218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.256182 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:36:05.290192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.268920 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:05.290192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.268972 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:05.290192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.289561 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:05.291227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.291199 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:05.291426 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.291396 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:05.386574 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.386529 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-openshift-ca" pod "45e4b2bc-b479-44cd-9532-f3ef6ae75452" (UID: "45e4b2bc-b479-44cd-9532-f3ef6ae75452"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:05.388399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.388253 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:05.390777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.353487 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-openshift-ca\") pod \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " Nov 01 22:36:05.436167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.412866 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:05.436167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.415435 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:05.436167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.415780 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:05.540808 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.522007 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-xks-ca\") pod \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " Nov 01 22:36:05.540808 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.522059 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e4b2bc-b479-44cd-9532-f3ef6ae75452-kube-api-access-lc572\" (UniqueName: \"kubernetes.io/projected/45e4b2bc-b479-44cd-9532-f3ef6ae75452-kube-api-access-lc572\") pod \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\" (UID: \"45e4b2bc-b479-44cd-9532-f3ef6ae75452\") " Nov 01 22:36:05.809260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.781512 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:05.836068 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.832340 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e4b2bc-b479-44cd-9532-f3ef6ae75452-kube-api-access-lc572" pod "45e4b2bc-b479-44cd-9532-f3ef6ae75452" (UID: "45e4b2bc-b479-44cd-9532-f3ef6ae75452"). InnerVolumeSpecName "kube-api-access-lc572". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:05.836068 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:05.832432 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:06.315138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.293878 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:06.331244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.320190 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:36:06.500146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.374013 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:36:06.502640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.382052 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:06.502947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.502912 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:06.503151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.430338 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:36:06.510578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.382106 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:06.510832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.510788 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:06.529688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.514357 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-xks-ca" pod "45e4b2bc-b479-44cd-9532-f3ef6ae75452" (UID: "45e4b2bc-b479-44cd-9532-f3ef6ae75452"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:06.844265 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.844215 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:36:06.849141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.845057 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:06.849141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.846495 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:36:06.879168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.856225 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:06.879168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.856451 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e4b2bc-b479-44cd-9532-f3ef6ae75452-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:06.879168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.856489 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc572\" (UniqueName: \"kubernetes.io/projected/45e4b2bc-b479-44cd-9532-f3ef6ae75452-kube-api-access-lc572\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:06.879168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.857691 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:06.890551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.885379 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:36:06.947745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.917740 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:06.947745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.924749 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:07.018000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:06.999232 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:36:07.041455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.040091 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:36:07.145212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.131541 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:07.145212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.131613 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:07.174644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.174604 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:07.174939 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.174903 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:07.357592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.357533 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:07.386356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.386314 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.393875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.393841 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph" Nov 01 22:36:07.566414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.566371 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:36:07.567331 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.567287 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:36:07.648475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.648429 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:36:07.700238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.700202 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-7sgls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:07.700487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.700457 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.870304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.870200 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.872387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.871253 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:07.872769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.871948 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:36:07.873126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.871991 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.873465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.872012 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.935732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.935698 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:07.935997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.935972 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.936568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.936500 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.954879 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.937685 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:07.955170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:07.954251 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:08.017074 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.015557 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Nov 01 22:36:08.017074 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.015643 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:36:08.062615 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.062561 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.154159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.154038 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:08.154431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.154398 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:08.230139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.228658 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:08.230139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.228714 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:08.353148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.343300 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Nov 01 22:36:08.385155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.385093 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.603138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.602222 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:36:08.603138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.603131 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.661137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.659188 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:36:08.708697 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:08.708661 6287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.142s" Nov 01 22:36:08.708999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.708981 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft"] Nov 01 22:36:08.710068 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.710042 6287 cpu_manager.go:414] "containerMap: removing container" podUID="45e4b2bc-b479-44cd-9532-f3ef6ae75452" containerName="revoke-keys" Nov 01 22:36:08.710216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.710200 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e4b2bc-b479-44cd-9532-f3ef6ae75452" containerName="revoke-keys" Nov 01 22:36:08.710588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.710571 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e4b2bc-b479-44cd-9532-f3ef6ae75452" containerName="revoke-keys" Nov 01 22:36:08.752403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.752372 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft"] Nov 01 22:36:08.752703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.752685 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:08.770834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.769422 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:36:08.770834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.770387 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:36:08.770834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.770457 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:36:08.770834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.770529 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:36:08.870534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.870444 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3595615b-9zjft\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:08.870915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.870890 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkk5g\" (UniqueName: \"kubernetes.io/projected/531d33c4-d487-41cb-bf66-99566c523e27-kube-api-access-mkk5g\") pod \"maas-api-revoke-sub-3595615b-9zjft\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:08.871173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.871101 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3595615b-9zjft\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:08.881135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.878388 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.881135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.878716 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.881135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.878780 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.881135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.879027 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:08.881135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.879285 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:36:08.881135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.879525 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:08.937894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.937854 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.955487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.955449 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:08.956470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.956442 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:09.002137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.997067 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkk5g\" (UniqueName: \"kubernetes.io/projected/531d33c4-d487-41cb-bf66-99566c523e27-kube-api-access-mkk5g\") pod \"maas-api-revoke-sub-3595615b-9zjft\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:09.002137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.997211 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3595615b-9zjft\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:09.002137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.997389 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3595615b-9zjft\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:09.002137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:08.998395 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3595615b-9zjft\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:09.032125 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.031278 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.052137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.044146 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3595615b-9zjft\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:09.063149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.061131 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkk5g\" (UniqueName: \"kubernetes.io/projected/531d33c4-d487-41cb-bf66-99566c523e27-kube-api-access-mkk5g\") pod \"maas-api-revoke-sub-3595615b-9zjft\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:09.094327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.094283 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.112504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.112462 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:09.112737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.112712 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:09.115102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.115076 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:36:09.366594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.359129 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:09.366594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.359204 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:09.366594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.357251 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.366594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.364166 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:09.409874 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.409842 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:36:09.452610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.452579 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:36:09.462059 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:09.454527 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:09.530029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.529986 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:09.636178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.622771 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt"] Nov 01 22:36:09.643356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.642963 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.654466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.654446 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:36:09.654639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.654628 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:36:09.654780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.654771 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:36:09.654931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.654920 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:36:09.656083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.656063 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:09.818934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.815537 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.823453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.823399 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:09.824354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.824323 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:36:09.831609 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.831550 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d1333ff-trjkt\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:09.831796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.831777 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d1333ff-trjkt\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:09.831865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.831829 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5qv\" (UniqueName: \"kubernetes.io/projected/d4ca8b98-2067-4131-bc57-4305ff2c5485-kube-api-access-2p5qv\") pod \"maas-api-revoke-sub-8d1333ff-trjkt\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:09.967875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.967766 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:10.010727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:09.998340 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:10.050995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.050968 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt"] Nov 01 22:36:10.053485 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.053350 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:36:10.054549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.054533 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:36:10.054741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.054716 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d1333ff-trjkt\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:10.054820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.054764 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5qv\" (UniqueName: \"kubernetes.io/projected/d4ca8b98-2067-4131-bc57-4305ff2c5485-kube-api-access-2p5qv\") pod \"maas-api-revoke-sub-8d1333ff-trjkt\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:10.055089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.055071 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d1333ff-trjkt\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:10.056182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.056162 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d1333ff-trjkt\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:10.087253 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.063132 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d1333ff-trjkt\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:10.088220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.088193 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:36:10.140200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.140154 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:36:10.140411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.139807 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:10.140625 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.140599 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:10.143313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.143275 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5qv\" (UniqueName: \"kubernetes.io/projected/d4ca8b98-2067-4131-bc57-4305ff2c5485-kube-api-access-2p5qv\") pod \"maas-api-revoke-sub-8d1333ff-trjkt\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:10.148486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.148460 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:36:10.164685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.164659 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:10.164818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.164706 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:10.170214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.170193 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:36:10.334607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.334529 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft"] Nov 01 22:36:10.342687 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:10.342659 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531d33c4_d487_41cb_bf66_99566c523e27.slice/crio-c944a91290220b43788e119efcfd33bef55bea068e20e0adf43be6f118f4bfc5 WatchSource:0}: Error finding container c944a91290220b43788e119efcfd33bef55bea068e20e0adf43be6f118f4bfc5: Status 404 returned error can't find the container with id c944a91290220b43788e119efcfd33bef55bea068e20e0adf43be6f118f4bfc5 Nov 01 22:36:10.441439 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.441407 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:10.647340 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.647183 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" event={"ID":"531d33c4-d487-41cb-bf66-99566c523e27","Type":"ContainerStarted","Data":"c944a91290220b43788e119efcfd33bef55bea068e20e0adf43be6f118f4bfc5"} Nov 01 22:36:10.935352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:10.935323 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt"] Nov 01 22:36:11.112210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:11.111715 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:11.112210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:11.111777 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:11.162839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:11.162791 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:11.163017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:11.162866 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:11.662036 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:11.661998 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" event={"ID":"531d33c4-d487-41cb-bf66-99566c523e27","Type":"ContainerStarted","Data":"1ef86b6d9d1d695cac490876d3d7c11ad7f581dfb7523ea31a43eaecfa1d5972"} Nov 01 22:36:11.664239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:11.664213 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" event={"ID":"d4ca8b98-2067-4131-bc57-4305ff2c5485","Type":"ContainerStarted","Data":"15bc607c4ed0a92fdd453cf3e007cd983195336fb801cade2d342ea04d8b1ec4"} Nov 01 22:36:11.664387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:11.664377 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" event={"ID":"d4ca8b98-2067-4131-bc57-4305ff2c5485","Type":"ContainerStarted","Data":"ade3fb846b537682ea2d867c5fb82b7433e9a157856ac692311926f1ca902ca2"} Nov 01 22:36:11.725141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:11.723839 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" podStartSLOduration=6.723820215 podStartE2EDuration="6.723820215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:11.694418863 +0000 UTC m=+784.342918101" watchObservedRunningTime="2026-11-01 22:36:11.723820215 +0000 UTC m=+784.372319454" Nov 01 22:36:11.733171 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:11.733092 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" podStartSLOduration=5.733078832 podStartE2EDuration="5.733078832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:11.724670229 +0000 UTC m=+784.373169465" watchObservedRunningTime="2026-11-01 22:36:11.733078832 +0000 UTC m=+784.381578059" Nov 01 22:36:12.134733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:12.134099 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:12.134733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:12.134222 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:12.162091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:12.162061 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:12.162244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:12.162138 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:12.722137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:12.722077 6287 generic.go:356] "Generic (PLEG): container finished" podID="d4ca8b98-2067-4131-bc57-4305ff2c5485" containerID="15bc607c4ed0a92fdd453cf3e007cd983195336fb801cade2d342ea04d8b1ec4" exitCode=0 Nov 01 22:36:12.722633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:12.722341 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" event={"ID":"d4ca8b98-2067-4131-bc57-4305ff2c5485","Type":"ContainerDied","Data":"15bc607c4ed0a92fdd453cf3e007cd983195336fb801cade2d342ea04d8b1ec4"} Nov 01 22:36:12.733401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:12.732976 6287 generic.go:356] "Generic (PLEG): container finished" podID="531d33c4-d487-41cb-bf66-99566c523e27" containerID="1ef86b6d9d1d695cac490876d3d7c11ad7f581dfb7523ea31a43eaecfa1d5972" exitCode=0 Nov 01 22:36:12.733401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:12.733075 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" event={"ID":"531d33c4-d487-41cb-bf66-99566c523e27","Type":"ContainerDied","Data":"1ef86b6d9d1d695cac490876d3d7c11ad7f581dfb7523ea31a43eaecfa1d5972"} Nov 01 22:36:13.109474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:13.109435 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 01 22:36:13.109648 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:13.109500 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 01 22:36:13.162577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:13.162545 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 01 22:36:13.162742 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:13.162598 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 01 22:36:14.132857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.118391 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:14.269751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.269718 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-xks-ca\") pod \"d4ca8b98-2067-4131-bc57-4305ff2c5485\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " Nov 01 22:36:14.270040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.269827 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-openshift-ca\") pod \"d4ca8b98-2067-4131-bc57-4305ff2c5485\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " Nov 01 22:36:14.270237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.270211 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-openshift-ca" pod "d4ca8b98-2067-4131-bc57-4305ff2c5485" (UID: "d4ca8b98-2067-4131-bc57-4305ff2c5485"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:14.270453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.270431 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ca8b98-2067-4131-bc57-4305ff2c5485-kube-api-access-2p5qv\" (UniqueName: \"kubernetes.io/projected/d4ca8b98-2067-4131-bc57-4305ff2c5485-kube-api-access-2p5qv\") pod \"d4ca8b98-2067-4131-bc57-4305ff2c5485\" (UID: \"d4ca8b98-2067-4131-bc57-4305ff2c5485\") " Nov 01 22:36:14.271488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.271459 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:14.284300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.284237 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ca8b98-2067-4131-bc57-4305ff2c5485-kube-api-access-2p5qv" pod "d4ca8b98-2067-4131-bc57-4305ff2c5485" (UID: "d4ca8b98-2067-4131-bc57-4305ff2c5485"). InnerVolumeSpecName "kube-api-access-2p5qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:14.288941 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.288874 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-xks-ca" pod "d4ca8b98-2067-4131-bc57-4305ff2c5485" (UID: "d4ca8b98-2067-4131-bc57-4305ff2c5485"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:14.301766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.301715 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:14.380176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.377024 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5qv\" (UniqueName: \"kubernetes.io/projected/d4ca8b98-2067-4131-bc57-4305ff2c5485-kube-api-access-2p5qv\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:14.380176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.377209 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ca8b98-2067-4131-bc57-4305ff2c5485-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:14.397421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.397382 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:14.401154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.401108 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:14.403172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.403151 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:36:14.478390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.477998 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-xks-ca\") pod \"531d33c4-d487-41cb-bf66-99566c523e27\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " Nov 01 22:36:14.478390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.478088 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531d33c4-d487-41cb-bf66-99566c523e27-kube-api-access-mkk5g\" (UniqueName: \"kubernetes.io/projected/531d33c4-d487-41cb-bf66-99566c523e27-kube-api-access-mkk5g\") pod \"531d33c4-d487-41cb-bf66-99566c523e27\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " Nov 01 22:36:14.478390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.478251 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-openshift-ca\") pod \"531d33c4-d487-41cb-bf66-99566c523e27\" (UID: \"531d33c4-d487-41cb-bf66-99566c523e27\") " Nov 01 22:36:14.489156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.479301 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-openshift-ca" pod "531d33c4-d487-41cb-bf66-99566c523e27" (UID: "531d33c4-d487-41cb-bf66-99566c523e27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:14.493673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.493650 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-xks-ca" pod "531d33c4-d487-41cb-bf66-99566c523e27" (UID: "531d33c4-d487-41cb-bf66-99566c523e27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:14.495612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.495585 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531d33c4-d487-41cb-bf66-99566c523e27-kube-api-access-mkk5g" pod "531d33c4-d487-41cb-bf66-99566c523e27" (UID: "531d33c4-d487-41cb-bf66-99566c523e27"). InnerVolumeSpecName "kube-api-access-mkk5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:14.560882 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.560786 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:14.561227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.561208 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:14.562215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.562192 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:36:14.582292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.582252 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:14.582292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.582283 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkk5g\" (UniqueName: \"kubernetes.io/projected/531d33c4-d487-41cb-bf66-99566c523e27-kube-api-access-mkk5g\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:14.582292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.582300 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531d33c4-d487-41cb-bf66-99566c523e27-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:14.788248 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.788192 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" Nov 01 22:36:14.788427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.788288 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft" event={"ID":"531d33c4-d487-41cb-bf66-99566c523e27","Type":"ContainerDied","Data":"c944a91290220b43788e119efcfd33bef55bea068e20e0adf43be6f118f4bfc5"} Nov 01 22:36:14.788427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.788345 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c944a91290220b43788e119efcfd33bef55bea068e20e0adf43be6f118f4bfc5" Nov 01 22:36:14.790747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.790725 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" Nov 01 22:36:14.791346 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.791316 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt" event={"ID":"d4ca8b98-2067-4131-bc57-4305ff2c5485","Type":"ContainerDied","Data":"ade3fb846b537682ea2d867c5fb82b7433e9a157856ac692311926f1ca902ca2"} Nov 01 22:36:14.791477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:14.791364 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ade3fb846b537682ea2d867c5fb82b7433e9a157856ac692311926f1ca902ca2" Nov 01 22:36:15.146717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.146681 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch"] Nov 01 22:36:15.147914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.147791 6287 cpu_manager.go:414] "containerMap: removing container" podUID="531d33c4-d487-41cb-bf66-99566c523e27" containerName="revoke-keys" Nov 01 22:36:15.147914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.147814 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531d33c4-d487-41cb-bf66-99566c523e27" containerName="revoke-keys" Nov 01 22:36:15.147914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.147832 6287 cpu_manager.go:414] "containerMap: removing container" podUID="d4ca8b98-2067-4131-bc57-4305ff2c5485" containerName="revoke-keys" Nov 01 22:36:15.147914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.147842 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ca8b98-2067-4131-bc57-4305ff2c5485" containerName="revoke-keys" Nov 01 22:36:15.148235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.148027 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="531d33c4-d487-41cb-bf66-99566c523e27" containerName="revoke-keys" Nov 01 22:36:15.148235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.148043 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ca8b98-2067-4131-bc57-4305ff2c5485" containerName="revoke-keys" Nov 01 22:36:15.152660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.152631 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.157410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.157391 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:36:15.157667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.157640 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:36:15.163502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.163472 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch"] Nov 01 22:36:15.189858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.189828 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:36:15.190083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.190062 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:36:15.295922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.295821 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lm5\" (UniqueName: \"kubernetes.io/projected/54f64e79-8ffd-45b1-82ae-54d0b562facd-kube-api-access-w2lm5\") pod \"maas-api-revoke-sub-bc91d391-t77ch\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.296108 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.296092 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc91d391-t77ch\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.296232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.296210 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc91d391-t77ch\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.397170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.397092 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc91d391-t77ch\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.397355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.397221 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc91d391-t77ch\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.397355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.397331 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lm5\" (UniqueName: \"kubernetes.io/projected/54f64e79-8ffd-45b1-82ae-54d0b562facd-kube-api-access-w2lm5\") pod \"maas-api-revoke-sub-bc91d391-t77ch\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.397876 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.397849 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc91d391-t77ch\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.399782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.399760 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc91d391-t77ch\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.405205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.405181 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lm5\" (UniqueName: \"kubernetes.io/projected/54f64e79-8ffd-45b1-82ae-54d0b562facd-kube-api-access-w2lm5\") pod \"maas-api-revoke-sub-bc91d391-t77ch\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.474506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.474480 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:15.603563 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:15.603474 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:36:15.755869 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:15.755838 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f64e79_8ffd_45b1_82ae_54d0b562facd.slice/crio-cd6cd094abf96eae230c5d6fc5f91f89afc4fdb7acc246ca7136960ac2d34adf WatchSource:0}: Error finding container cd6cd094abf96eae230c5d6fc5f91f89afc4fdb7acc246ca7136960ac2d34adf: Status 404 returned error can't find the container with id cd6cd094abf96eae230c5d6fc5f91f89afc4fdb7acc246ca7136960ac2d34adf Nov 01 22:36:15.758017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.757992 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch"] Nov 01 22:36:15.801721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:15.801697 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" event={"ID":"54f64e79-8ffd-45b1-82ae-54d0b562facd","Type":"ContainerStarted","Data":"cd6cd094abf96eae230c5d6fc5f91f89afc4fdb7acc246ca7136960ac2d34adf"} Nov 01 22:36:16.817226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:16.817182 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" event={"ID":"54f64e79-8ffd-45b1-82ae-54d0b562facd","Type":"ContainerStarted","Data":"0b8a427e403ff9b492122a4306bd888313cb62306dd5e5f616a5cb19d8929db0"} Nov 01 22:36:16.835878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:16.835829 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" podStartSLOduration=1.835816369 podStartE2EDuration="1.835816369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:16.832936086 +0000 UTC m=+789.481435321" watchObservedRunningTime="2026-11-01 22:36:16.835816369 +0000 UTC m=+789.484315648" Nov 01 22:36:17.828042 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:17.828002 6287 generic.go:356] "Generic (PLEG): container finished" podID="54f64e79-8ffd-45b1-82ae-54d0b562facd" containerID="0b8a427e403ff9b492122a4306bd888313cb62306dd5e5f616a5cb19d8929db0" exitCode=0 Nov 01 22:36:17.828511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:17.828090 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" event={"ID":"54f64e79-8ffd-45b1-82ae-54d0b562facd","Type":"ContainerDied","Data":"0b8a427e403ff9b492122a4306bd888313cb62306dd5e5f616a5cb19d8929db0"} Nov 01 22:36:18.682275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.682241 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6"] Nov 01 22:36:18.687685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.687659 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:18.695177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.695102 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6"] Nov 01 22:36:18.700727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.699784 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fd214936\"" Nov 01 22:36:18.783080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.783045 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-plugins-config-volume\") pod \"payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6\" (UID: \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\") " pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:18.783269 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.783173 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2t4r\" (UniqueName: \"kubernetes.io/projected/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-kube-api-access-w2t4r\") pod \"payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6\" (UID: \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\") " pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:18.922135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.917410 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-plugins-config-volume\") pod \"payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6\" (UID: \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\") " pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:18.922135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.917514 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2t4r\" (UniqueName: \"kubernetes.io/projected/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-kube-api-access-w2t4r\") pod \"payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6\" (UID: \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\") " pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:18.922135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.918783 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-plugins-config-volume\") pod \"payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6\" (UID: \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\") " pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:18.970771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:18.970697 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2t4r\" (UniqueName: \"kubernetes.io/projected/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-kube-api-access-w2t4r\") pod \"payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6\" (UID: \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\") " pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:19.015759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.015727 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:19.265357 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.265325 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:19.315790 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:19.315036 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:19.344164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.342904 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-xks-ca\") pod \"54f64e79-8ffd-45b1-82ae-54d0b562facd\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " Nov 01 22:36:19.344164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.342959 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-openshift-ca\") pod \"54f64e79-8ffd-45b1-82ae-54d0b562facd\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " Nov 01 22:36:19.344164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.343397 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f64e79-8ffd-45b1-82ae-54d0b562facd-kube-api-access-w2lm5\" (UniqueName: \"kubernetes.io/projected/54f64e79-8ffd-45b1-82ae-54d0b562facd-kube-api-access-w2lm5\") pod \"54f64e79-8ffd-45b1-82ae-54d0b562facd\" (UID: \"54f64e79-8ffd-45b1-82ae-54d0b562facd\") " Nov 01 22:36:19.345139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.345091 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-openshift-ca" pod "54f64e79-8ffd-45b1-82ae-54d0b562facd" (UID: "54f64e79-8ffd-45b1-82ae-54d0b562facd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:19.346829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.346804 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-xks-ca" pod "54f64e79-8ffd-45b1-82ae-54d0b562facd" (UID: "54f64e79-8ffd-45b1-82ae-54d0b562facd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:19.405207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.405144 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f64e79-8ffd-45b1-82ae-54d0b562facd-kube-api-access-w2lm5" pod "54f64e79-8ffd-45b1-82ae-54d0b562facd" (UID: "54f64e79-8ffd-45b1-82ae-54d0b562facd"). InnerVolumeSpecName "kube-api-access-w2lm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:19.445521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.445486 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2lm5\" (UniqueName: \"kubernetes.io/projected/54f64e79-8ffd-45b1-82ae-54d0b562facd-kube-api-access-w2lm5\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:19.445521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.445517 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:19.445761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.445534 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f64e79-8ffd-45b1-82ae-54d0b562facd-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:19.517969 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:19.517912 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4ca24e_dbb4_4d92_a60f_412aae7f8e71.slice/crio-f0139910778fd5cb123e1b950a773226b49c31a895bec4f4b4940e9360d10179 WatchSource:0}: Error finding container f0139910778fd5cb123e1b950a773226b49c31a895bec4f4b4940e9360d10179: Status 404 returned error can't find the container with id f0139910778fd5cb123e1b950a773226b49c31a895bec4f4b4940e9360d10179 Nov 01 22:36:19.520212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.520184 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6"] Nov 01 22:36:19.562974 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.562942 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs"] Nov 01 22:36:19.563851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.563828 6287 cpu_manager.go:414] "containerMap: removing container" podUID="54f64e79-8ffd-45b1-82ae-54d0b562facd" containerName="revoke-keys" Nov 01 22:36:19.563851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.563848 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f64e79-8ffd-45b1-82ae-54d0b562facd" containerName="revoke-keys" Nov 01 22:36:19.564006 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.563974 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f64e79-8ffd-45b1-82ae-54d0b562facd" containerName="revoke-keys" Nov 01 22:36:19.617747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.617586 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs"] Nov 01 22:36:19.617747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.617706 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.649619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.649540 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zq9f\" (UniqueName: \"kubernetes.io/projected/922d9d62-2412-4c75-aa1d-a6931bea1e37-kube-api-access-9zq9f\") pod \"maas-api-revoke-sub-4a093c28-4fdxs\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.650027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.649979 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a093c28-4fdxs\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.650146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.650101 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a093c28-4fdxs\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.751210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.751172 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a093c28-4fdxs\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.751400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.751280 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a093c28-4fdxs\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.751509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.751494 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zq9f\" (UniqueName: \"kubernetes.io/projected/922d9d62-2412-4c75-aa1d-a6931bea1e37-kube-api-access-9zq9f\") pod \"maas-api-revoke-sub-4a093c28-4fdxs\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.751928 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.751895 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a093c28-4fdxs\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.754414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.754386 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a093c28-4fdxs\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.793186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.790191 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zq9f\" (UniqueName: \"kubernetes.io/projected/922d9d62-2412-4c75-aa1d-a6931bea1e37-kube-api-access-9zq9f\") pod \"maas-api-revoke-sub-4a093c28-4fdxs\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:19.849776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.849746 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" Nov 01 22:36:19.849776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.849749 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch" event={"ID":"54f64e79-8ffd-45b1-82ae-54d0b562facd","Type":"ContainerDied","Data":"cd6cd094abf96eae230c5d6fc5f91f89afc4fdb7acc246ca7136960ac2d34adf"} Nov 01 22:36:19.850045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.849815 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd6cd094abf96eae230c5d6fc5f91f89afc4fdb7acc246ca7136960ac2d34adf" Nov 01 22:36:19.851923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.851896 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" event={"ID":"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71","Type":"ContainerStarted","Data":"f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514"} Nov 01 22:36:19.852052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.851940 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" event={"ID":"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71","Type":"ContainerStarted","Data":"f0139910778fd5cb123e1b950a773226b49c31a895bec4f4b4940e9360d10179"} Nov 01 22:36:19.852456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.852438 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:19.880650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.880602 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" podStartSLOduration=1.880586041 podStartE2EDuration="1.880586041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:19.874620545 +0000 UTC m=+792.523119772" watchObservedRunningTime="2026-11-01 22:36:19.880586041 +0000 UTC m=+792.529085279" Nov 01 22:36:19.945054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:19.945017 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:20.254712 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:20.254676 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922d9d62_2412_4c75_aa1d_a6931bea1e37.slice/crio-8f12b68c54acec455b4cbf98ad87ce70cf183ce54762384cb0ee5558d8ad6f9e WatchSource:0}: Error finding container 8f12b68c54acec455b4cbf98ad87ce70cf183ce54762384cb0ee5558d8ad6f9e: Status 404 returned error can't find the container with id 8f12b68c54acec455b4cbf98ad87ce70cf183ce54762384cb0ee5558d8ad6f9e Nov 01 22:36:20.257040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:20.257016 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs"] Nov 01 22:36:20.864904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:20.864800 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" event={"ID":"922d9d62-2412-4c75-aa1d-a6931bea1e37","Type":"ContainerStarted","Data":"0332f72a5543edee7ec77133ae9a457aec76b7949205f414f4e3b04d8a6ef647"} Nov 01 22:36:20.864904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:20.864870 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" event={"ID":"922d9d62-2412-4c75-aa1d-a6931bea1e37","Type":"ContainerStarted","Data":"8f12b68c54acec455b4cbf98ad87ce70cf183ce54762384cb0ee5558d8ad6f9e"} Nov 01 22:36:20.896365 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:20.896195 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" podStartSLOduration=1.896170986 podStartE2EDuration="1.896170986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:20.886320963 +0000 UTC m=+793.534820199" watchObservedRunningTime="2026-11-01 22:36:20.896170986 +0000 UTC m=+793.544670225" Nov 01 22:36:21.639748 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.639712 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz"] Nov 01 22:36:21.656476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.656443 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.659880 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.659853 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz"] Nov 01 22:36:21.687822 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.687782 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0beb384-77xsz\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.689149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.687934 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlck2\" (UniqueName: \"kubernetes.io/projected/176b9369-fceb-4e6b-9807-9d5cf2a21754-kube-api-access-wlck2\") pod \"maas-api-revoke-sub-c0beb384-77xsz\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.689149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.688045 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0beb384-77xsz\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.792215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.792176 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlck2\" (UniqueName: \"kubernetes.io/projected/176b9369-fceb-4e6b-9807-9d5cf2a21754-kube-api-access-wlck2\") pod \"maas-api-revoke-sub-c0beb384-77xsz\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.792385 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.792280 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0beb384-77xsz\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.792457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.792445 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0beb384-77xsz\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.793077 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.793059 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0beb384-77xsz\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.794762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.794740 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0beb384-77xsz\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.816578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.816551 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlck2\" (UniqueName: \"kubernetes.io/projected/176b9369-fceb-4e6b-9807-9d5cf2a21754-kube-api-access-wlck2\") pod \"maas-api-revoke-sub-c0beb384-77xsz\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:21.876486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.876457 6287 generic.go:356] "Generic (PLEG): container finished" podID="922d9d62-2412-4c75-aa1d-a6931bea1e37" containerID="0332f72a5543edee7ec77133ae9a457aec76b7949205f414f4e3b04d8a6ef647" exitCode=0 Nov 01 22:36:21.876652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.876639 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" event={"ID":"922d9d62-2412-4c75-aa1d-a6931bea1e37","Type":"ContainerDied","Data":"0332f72a5543edee7ec77133ae9a457aec76b7949205f414f4e3b04d8a6ef647"} Nov 01 22:36:21.977203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:21.977108 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:22.260444 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:22.260399 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176b9369_fceb_4e6b_9807_9d5cf2a21754.slice/crio-5cd35bd7178cb05e97ce69f5a370aa59b4ebc56aa03f40fdee48e8a7a91d3df2 WatchSource:0}: Error finding container 5cd35bd7178cb05e97ce69f5a370aa59b4ebc56aa03f40fdee48e8a7a91d3df2: Status 404 returned error can't find the container with id 5cd35bd7178cb05e97ce69f5a370aa59b4ebc56aa03f40fdee48e8a7a91d3df2 Nov 01 22:36:22.263903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:22.263875 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz"] Nov 01 22:36:22.889175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:22.889070 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" event={"ID":"176b9369-fceb-4e6b-9807-9d5cf2a21754","Type":"ContainerStarted","Data":"a8a14b8e1e4030f016cc2d2879d7bbab0be07fd5b081ca409ed7f5a8e66b1bbe"} Nov 01 22:36:22.889175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:22.889175 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" event={"ID":"176b9369-fceb-4e6b-9807-9d5cf2a21754","Type":"ContainerStarted","Data":"5cd35bd7178cb05e97ce69f5a370aa59b4ebc56aa03f40fdee48e8a7a91d3df2"} Nov 01 22:36:22.913906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:22.913854 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" podStartSLOduration=1.913839442 podStartE2EDuration="1.913839442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:22.90877707 +0000 UTC m=+795.557276305" watchObservedRunningTime="2026-11-01 22:36:22.913839442 +0000 UTC m=+795.562338676" Nov 01 22:36:23.053517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.053486 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:23.119251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.119210 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-openshift-ca\") pod \"922d9d62-2412-4c75-aa1d-a6931bea1e37\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " Nov 01 22:36:23.119435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.119292 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922d9d62-2412-4c75-aa1d-a6931bea1e37-kube-api-access-9zq9f\" (UniqueName: \"kubernetes.io/projected/922d9d62-2412-4c75-aa1d-a6931bea1e37-kube-api-access-9zq9f\") pod \"922d9d62-2412-4c75-aa1d-a6931bea1e37\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " Nov 01 22:36:23.119592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.119571 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-xks-ca\") pod \"922d9d62-2412-4c75-aa1d-a6931bea1e37\" (UID: \"922d9d62-2412-4c75-aa1d-a6931bea1e37\") " Nov 01 22:36:23.120771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.120727 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-openshift-ca" pod "922d9d62-2412-4c75-aa1d-a6931bea1e37" (UID: "922d9d62-2412-4c75-aa1d-a6931bea1e37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:23.131236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.122533 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-xks-ca" pod "922d9d62-2412-4c75-aa1d-a6931bea1e37" (UID: "922d9d62-2412-4c75-aa1d-a6931bea1e37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:23.131236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.122688 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922d9d62-2412-4c75-aa1d-a6931bea1e37-kube-api-access-9zq9f" pod "922d9d62-2412-4c75-aa1d-a6931bea1e37" (UID: "922d9d62-2412-4c75-aa1d-a6931bea1e37"). InnerVolumeSpecName "kube-api-access-9zq9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:23.131236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.123202 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:23.131236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.123221 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922d9d62-2412-4c75-aa1d-a6931bea1e37-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:23.131236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.123231 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zq9f\" (UniqueName: \"kubernetes.io/projected/922d9d62-2412-4c75-aa1d-a6931bea1e37-kube-api-access-9zq9f\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:23.925470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.925437 6287 generic.go:356] "Generic (PLEG): container finished" podID="176b9369-fceb-4e6b-9807-9d5cf2a21754" containerID="a8a14b8e1e4030f016cc2d2879d7bbab0be07fd5b081ca409ed7f5a8e66b1bbe" exitCode=0 Nov 01 22:36:23.925947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.925525 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" event={"ID":"176b9369-fceb-4e6b-9807-9d5cf2a21754","Type":"ContainerDied","Data":"a8a14b8e1e4030f016cc2d2879d7bbab0be07fd5b081ca409ed7f5a8e66b1bbe"} Nov 01 22:36:23.927398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.927377 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" Nov 01 22:36:23.927517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.927389 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs" event={"ID":"922d9d62-2412-4c75-aa1d-a6931bea1e37","Type":"ContainerDied","Data":"8f12b68c54acec455b4cbf98ad87ce70cf183ce54762384cb0ee5558d8ad6f9e"} Nov 01 22:36:23.927517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:23.927439 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f12b68c54acec455b4cbf98ad87ce70cf183ce54762384cb0ee5558d8ad6f9e" Nov 01 22:36:24.371905 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:24.371872 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:25.170100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.170065 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:25.256016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.255976 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-openshift-ca\") pod \"176b9369-fceb-4e6b-9807-9d5cf2a21754\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " Nov 01 22:36:25.256228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.256192 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176b9369-fceb-4e6b-9807-9d5cf2a21754-kube-api-access-wlck2\" (UniqueName: \"kubernetes.io/projected/176b9369-fceb-4e6b-9807-9d5cf2a21754-kube-api-access-wlck2\") pod \"176b9369-fceb-4e6b-9807-9d5cf2a21754\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " Nov 01 22:36:25.256299 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.256268 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-xks-ca\") pod \"176b9369-fceb-4e6b-9807-9d5cf2a21754\" (UID: \"176b9369-fceb-4e6b-9807-9d5cf2a21754\") " Nov 01 22:36:25.257237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.257198 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-openshift-ca" pod "176b9369-fceb-4e6b-9807-9d5cf2a21754" (UID: "176b9369-fceb-4e6b-9807-9d5cf2a21754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:25.257637 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.257611 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:25.259979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.259949 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-xks-ca" pod "176b9369-fceb-4e6b-9807-9d5cf2a21754" (UID: "176b9369-fceb-4e6b-9807-9d5cf2a21754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:25.271497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.271463 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176b9369-fceb-4e6b-9807-9d5cf2a21754-kube-api-access-wlck2" pod "176b9369-fceb-4e6b-9807-9d5cf2a21754" (UID: "176b9369-fceb-4e6b-9807-9d5cf2a21754"). InnerVolumeSpecName "kube-api-access-wlck2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:25.359186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.359159 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlck2\" (UniqueName: \"kubernetes.io/projected/176b9369-fceb-4e6b-9807-9d5cf2a21754-kube-api-access-wlck2\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:25.359186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.359184 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/176b9369-fceb-4e6b-9807-9d5cf2a21754-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:25.403325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.403293 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z"] Nov 01 22:36:25.404030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.404013 6287 cpu_manager.go:414] "containerMap: removing container" podUID="176b9369-fceb-4e6b-9807-9d5cf2a21754" containerName="revoke-keys" Nov 01 22:36:25.404030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.404029 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176b9369-fceb-4e6b-9807-9d5cf2a21754" containerName="revoke-keys" Nov 01 22:36:25.404192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.404043 6287 cpu_manager.go:414] "containerMap: removing container" podUID="922d9d62-2412-4c75-aa1d-a6931bea1e37" containerName="revoke-keys" Nov 01 22:36:25.404192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.404048 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922d9d62-2412-4c75-aa1d-a6931bea1e37" containerName="revoke-keys" Nov 01 22:36:25.404297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.404218 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="176b9369-fceb-4e6b-9807-9d5cf2a21754" containerName="revoke-keys" Nov 01 22:36:25.404297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.404229 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="922d9d62-2412-4c75-aa1d-a6931bea1e37" containerName="revoke-keys" Nov 01 22:36:25.409161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.408382 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.418564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.418537 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z"] Nov 01 22:36:25.562711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.562633 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82tkb\" (UniqueName: \"kubernetes.io/projected/04aa83a3-acba-4a30-b195-f898e52d3825-kube-api-access-82tkb\") pod \"maas-api-revoke-sub-b4e3c26a-lvp2z\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.562843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.562777 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e3c26a-lvp2z\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.562843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.562812 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e3c26a-lvp2z\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.668614 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.667302 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e3c26a-lvp2z\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.668614 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.667384 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e3c26a-lvp2z\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.668614 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.667682 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82tkb\" (UniqueName: \"kubernetes.io/projected/04aa83a3-acba-4a30-b195-f898e52d3825-kube-api-access-82tkb\") pod \"maas-api-revoke-sub-b4e3c26a-lvp2z\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.669685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.669656 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e3c26a-lvp2z\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.685970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.684217 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82tkb\" (UniqueName: \"kubernetes.io/projected/04aa83a3-acba-4a30-b195-f898e52d3825-kube-api-access-82tkb\") pod \"maas-api-revoke-sub-b4e3c26a-lvp2z\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.685970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.684614 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e3c26a-lvp2z\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.727158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.727097 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:25.952093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.952066 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" Nov 01 22:36:25.952309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.952143 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz" event={"ID":"176b9369-fceb-4e6b-9807-9d5cf2a21754","Type":"ContainerDied","Data":"5cd35bd7178cb05e97ce69f5a370aa59b4ebc56aa03f40fdee48e8a7a91d3df2"} Nov 01 22:36:25.952309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:25.952202 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cd35bd7178cb05e97ce69f5a370aa59b4ebc56aa03f40fdee48e8a7a91d3df2" Nov 01 22:36:26.041040 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:26.041012 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04aa83a3_acba_4a30_b195_f898e52d3825.slice/crio-6a411f88e7f0dcfc521fdb6fd0845a71196c71818bc3c5b76fce4e708d1c42fe WatchSource:0}: Error finding container 6a411f88e7f0dcfc521fdb6fd0845a71196c71818bc3c5b76fce4e708d1c42fe: Status 404 returned error can't find the container with id 6a411f88e7f0dcfc521fdb6fd0845a71196c71818bc3c5b76fce4e708d1c42fe Nov 01 22:36:26.044102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:26.044071 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z"] Nov 01 22:36:26.963089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:26.963053 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" event={"ID":"04aa83a3-acba-4a30-b195-f898e52d3825","Type":"ContainerStarted","Data":"1a6185c9900781d0cd24eb42595533cc14e4ffc4b147beb479b25160ab5b951c"} Nov 01 22:36:26.963511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:26.963123 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" event={"ID":"04aa83a3-acba-4a30-b195-f898e52d3825","Type":"ContainerStarted","Data":"6a411f88e7f0dcfc521fdb6fd0845a71196c71818bc3c5b76fce4e708d1c42fe"} Nov 01 22:36:26.985394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:26.985346 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" podStartSLOduration=1.9853304170000001 podStartE2EDuration="1.985330417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:26.980962334 +0000 UTC m=+799.629461571" watchObservedRunningTime="2026-11-01 22:36:26.985330417 +0000 UTC m=+799.633829652" Nov 01 22:36:27.976058 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:27.976024 6287 generic.go:356] "Generic (PLEG): container finished" podID="04aa83a3-acba-4a30-b195-f898e52d3825" containerID="1a6185c9900781d0cd24eb42595533cc14e4ffc4b147beb479b25160ab5b951c" exitCode=0 Nov 01 22:36:27.976484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:27.976083 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" event={"ID":"04aa83a3-acba-4a30-b195-f898e52d3825","Type":"ContainerDied","Data":"1a6185c9900781d0cd24eb42595533cc14e4ffc4b147beb479b25160ab5b951c"} Nov 01 22:36:29.212652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.212627 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:29.223888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.223865 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md"] Nov 01 22:36:29.224537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.224523 6287 cpu_manager.go:414] "containerMap: removing container" podUID="04aa83a3-acba-4a30-b195-f898e52d3825" containerName="revoke-keys" Nov 01 22:36:29.224578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.224538 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04aa83a3-acba-4a30-b195-f898e52d3825" containerName="revoke-keys" Nov 01 22:36:29.224706 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.224694 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="04aa83a3-acba-4a30-b195-f898e52d3825" containerName="revoke-keys" Nov 01 22:36:29.228485 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.228465 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.245783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.245752 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.246165 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.246138 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q669t\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-kube-api-access-q669t\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.246497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.246474 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.246638 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.246611 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.246788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.246769 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.246943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.246926 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.247140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.247103 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.247454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.247430 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.247616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.247597 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.261671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.261642 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md"] Nov 01 22:36:29.339201 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:29.339109 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:29.348603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.348578 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-openshift-ca\") pod \"04aa83a3-acba-4a30-b195-f898e52d3825\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " Nov 01 22:36:29.348730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.348646 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04aa83a3-acba-4a30-b195-f898e52d3825-kube-api-access-82tkb\" (UniqueName: \"kubernetes.io/projected/04aa83a3-acba-4a30-b195-f898e52d3825-kube-api-access-82tkb\") pod \"04aa83a3-acba-4a30-b195-f898e52d3825\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " Nov 01 22:36:29.348843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.348823 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-xks-ca\") pod \"04aa83a3-acba-4a30-b195-f898e52d3825\" (UID: \"04aa83a3-acba-4a30-b195-f898e52d3825\") " Nov 01 22:36:29.348980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.348959 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-openshift-ca" pod "04aa83a3-acba-4a30-b195-f898e52d3825" (UID: "04aa83a3-acba-4a30-b195-f898e52d3825"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:29.349281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349260 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349288 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349337 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349366 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349434 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349540 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349580 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349652 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349667 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-credential-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349694 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q669t\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-kube-api-access-q669t\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349806 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-data\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.349881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.349765 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-certs\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.350261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.350238 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-socket\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.350352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.350244 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:29.350595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.350569 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istiod-ca-cert\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.352427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.352398 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aa83a3-acba-4a30-b195-f898e52d3825-kube-api-access-82tkb" pod "04aa83a3-acba-4a30-b195-f898e52d3825" (UID: "04aa83a3-acba-4a30-b195-f898e52d3825"). InnerVolumeSpecName "kube-api-access-82tkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:29.352507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.352474 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-xks-ca" pod "04aa83a3-acba-4a30-b195-f898e52d3825" (UID: "04aa83a3-acba-4a30-b195-f898e52d3825"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:29.352895 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.352864 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-envoy\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.352895 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.352875 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-podinfo\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.358357 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.358331 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-token\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.359225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.359203 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q669t\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-kube-api-access-q669t\") pod \"e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.451954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.451919 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82tkb\" (UniqueName: \"kubernetes.io/projected/04aa83a3-acba-4a30-b195-f898e52d3825-kube-api-access-82tkb\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:29.451954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.451950 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04aa83a3-acba-4a30-b195-f898e52d3825-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:29.563057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.563027 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:29.776153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.776105 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg"] Nov 01 22:36:29.782202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.782175 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:29.789340 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.789236 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg"] Nov 01 22:36:29.855081 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:29.855049 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfe9406_1e6e_4370_8b03_2b5cbb4793ae.slice/crio-601fe584f7bcfc8e54884769f40b30711cc9b08a70c1db90d347970f1933d62e WatchSource:0}: Error finding container 601fe584f7bcfc8e54884769f40b30711cc9b08a70c1db90d347970f1933d62e: Status 404 returned error can't find the container with id 601fe584f7bcfc8e54884769f40b30711cc9b08a70c1db90d347970f1933d62e Nov 01 22:36:29.857538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.857510 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fa36466-9qdhg\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:29.857678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.857657 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fa36466-9qdhg\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:29.857862 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.857843 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tcx\" (UniqueName: \"kubernetes.io/projected/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-kube-api-access-b6tcx\") pod \"maas-api-revoke-sub-5fa36466-9qdhg\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:29.857912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.857842 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:29.857954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.857919 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:29.857954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.857949 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:29.859359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.859338 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md"] Nov 01 22:36:29.967807 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.959355 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fa36466-9qdhg\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:29.967807 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.959525 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fa36466-9qdhg\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:29.967807 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.959785 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tcx\" (UniqueName: \"kubernetes.io/projected/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-kube-api-access-b6tcx\") pod \"maas-api-revoke-sub-5fa36466-9qdhg\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:29.967807 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.961047 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fa36466-9qdhg\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:29.978246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.976221 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fa36466-9qdhg\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:29.981680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:29.981222 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tcx\" (UniqueName: \"kubernetes.io/projected/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-kube-api-access-b6tcx\") pod \"maas-api-revoke-sub-5fa36466-9qdhg\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:30.051710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.051677 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" Nov 01 22:36:30.051901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.051723 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z" event={"ID":"04aa83a3-acba-4a30-b195-f898e52d3825","Type":"ContainerDied","Data":"6a411f88e7f0dcfc521fdb6fd0845a71196c71818bc3c5b76fce4e708d1c42fe"} Nov 01 22:36:30.051901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.051783 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a411f88e7f0dcfc521fdb6fd0845a71196c71818bc3c5b76fce4e708d1c42fe" Nov 01 22:36:30.054865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.054078 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" event={"ID":"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae","Type":"ContainerStarted","Data":"d000f5344a67946cdc3400927645ec601966e832829e963d97b6733f0c851ac1"} Nov 01 22:36:30.054865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.054211 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" event={"ID":"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae","Type":"ContainerStarted","Data":"601fe584f7bcfc8e54884769f40b30711cc9b08a70c1db90d347970f1933d62e"} Nov 01 22:36:30.085202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.085153 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podStartSLOduration=1.085139195 podStartE2EDuration="1.085139195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:30.077169128 +0000 UTC m=+802.725668364" watchObservedRunningTime="2026-11-01 22:36:30.085139195 +0000 UTC m=+802.733638430" Nov 01 22:36:30.106164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.106096 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:30.191045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.191017 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v"] Nov 01 22:36:30.199812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.198983 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.219889 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.219863 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v"] Nov 01 22:36:30.265211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.265178 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0500c01f-plx5v\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.265357 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.265317 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8v9j\" (UniqueName: \"kubernetes.io/projected/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-kube-api-access-h8v9j\") pod \"maas-api-revoke-keys-e2e-ait-0500c01f-plx5v\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.265577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.265530 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0500c01f-plx5v\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.366657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.366624 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0500c01f-plx5v\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.366814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.366770 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0500c01f-plx5v\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.366948 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.366928 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8v9j\" (UniqueName: \"kubernetes.io/projected/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-kube-api-access-h8v9j\") pod \"maas-api-revoke-keys-e2e-ait-0500c01f-plx5v\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.367534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.367514 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0500c01f-plx5v\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.369107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.369085 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0500c01f-plx5v\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.375499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.375478 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8v9j\" (UniqueName: \"kubernetes.io/projected/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-kube-api-access-h8v9j\") pod \"maas-api-revoke-keys-e2e-ait-0500c01f-plx5v\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.399861 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:30.399835 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c7006c_c170_4188_a7b5_27bfbbb6c82a.slice/crio-dd7cfb014dd61dbaa25be8922dea70b6f67b5a45fd427cb8da3edf8c3ace6abe WatchSource:0}: Error finding container dd7cfb014dd61dbaa25be8922dea70b6f67b5a45fd427cb8da3edf8c3ace6abe: Status 404 returned error can't find the container with id dd7cfb014dd61dbaa25be8922dea70b6f67b5a45fd427cb8da3edf8c3ace6abe Nov 01 22:36:30.404708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.404676 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg"] Nov 01 22:36:30.533750 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.533720 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:30.564195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.564163 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:30.565855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.565831 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 22:36:30.565960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.565883 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 22:36:30.847488 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:30.847458 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6aadea_133d_46e9_a3ba_3b8b81633a0a.slice/crio-b7b5bd75fd0e2cb7a61d995f76af230c891c582d8d405fa42fd77dc26f7123df WatchSource:0}: Error finding container b7b5bd75fd0e2cb7a61d995f76af230c891c582d8d405fa42fd77dc26f7123df: Status 404 returned error can't find the container with id b7b5bd75fd0e2cb7a61d995f76af230c891c582d8d405fa42fd77dc26f7123df Nov 01 22:36:30.849938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.849916 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v"] Nov 01 22:36:30.868079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:30.868054 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:36:31.147295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:31.147249 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" event={"ID":"7c6aadea-133d-46e9-a3ba-3b8b81633a0a","Type":"ContainerStarted","Data":"b7b5bd75fd0e2cb7a61d995f76af230c891c582d8d405fa42fd77dc26f7123df"} Nov 01 22:36:31.152316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:31.152281 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" event={"ID":"b3c7006c-c170-4188-a7b5-27bfbbb6c82a","Type":"ContainerStarted","Data":"796164a7463f141c90d64858814c26bc4fcd54db2f810749bb8ef748015f055d"} Nov 01 22:36:31.152460 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:31.152332 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" event={"ID":"b3c7006c-c170-4188-a7b5-27bfbbb6c82a","Type":"ContainerStarted","Data":"dd7cfb014dd61dbaa25be8922dea70b6f67b5a45fd427cb8da3edf8c3ace6abe"} Nov 01 22:36:31.205135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:31.204686 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" podStartSLOduration=2.204663475 podStartE2EDuration="2.204663475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:31.192536402 +0000 UTC m=+803.841035638" watchObservedRunningTime="2026-11-01 22:36:31.204663475 +0000 UTC m=+803.853162712" Nov 01 22:36:31.564423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:31.564389 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 22:36:31.564786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:31.564453 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 22:36:31.959796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:31.959733 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd"] Nov 01 22:36:31.966213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:31.966174 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.051281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.047576 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd"] Nov 01 22:36:32.053810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.052552 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9cc6c56-nv5vd\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.053810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.052611 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9cc6c56-nv5vd\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.053810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.052819 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5gbz\" (UniqueName: \"kubernetes.io/projected/078d57ed-1850-4485-b27f-b38514f5325a-kube-api-access-l5gbz\") pod \"maas-api-revoke-sub-f9cc6c56-nv5vd\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.153971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.153944 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gbz\" (UniqueName: \"kubernetes.io/projected/078d57ed-1850-4485-b27f-b38514f5325a-kube-api-access-l5gbz\") pod \"maas-api-revoke-sub-f9cc6c56-nv5vd\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.154135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.154094 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9cc6c56-nv5vd\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.154211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.154138 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9cc6c56-nv5vd\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.154821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.154799 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9cc6c56-nv5vd\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.156385 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.156365 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9cc6c56-nv5vd\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.162089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.162064 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gbz\" (UniqueName: \"kubernetes.io/projected/078d57ed-1850-4485-b27f-b38514f5325a-kube-api-access-l5gbz\") pod \"maas-api-revoke-sub-f9cc6c56-nv5vd\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.163338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.163311 6287 generic.go:356] "Generic (PLEG): container finished" podID="7c6aadea-133d-46e9-a3ba-3b8b81633a0a" containerID="46392a08783900b7a4ad6237264826b0cca0ddaa57ed33f7aa971ecd0713094c" exitCode=0 Nov 01 22:36:32.163425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.163389 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" event={"ID":"7c6aadea-133d-46e9-a3ba-3b8b81633a0a","Type":"ContainerDied","Data":"46392a08783900b7a4ad6237264826b0cca0ddaa57ed33f7aa971ecd0713094c"} Nov 01 22:36:32.165199 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.165179 6287 generic.go:356] "Generic (PLEG): container finished" podID="b3c7006c-c170-4188-a7b5-27bfbbb6c82a" containerID="796164a7463f141c90d64858814c26bc4fcd54db2f810749bb8ef748015f055d" exitCode=0 Nov 01 22:36:32.165277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.165235 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" event={"ID":"b3c7006c-c170-4188-a7b5-27bfbbb6c82a","Type":"ContainerDied","Data":"796164a7463f141c90d64858814c26bc4fcd54db2f810749bb8ef748015f055d"} Nov 01 22:36:32.364702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.364664 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:32.564437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.564403 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 22:36:32.564824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.564453 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 22:36:32.701368 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:32.701332 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078d57ed_1850_4485_b27f_b38514f5325a.slice/crio-e224f604d7c255c6fb131273e9945403bfe5be95e51fa5abcc93306346bf69ea WatchSource:0}: Error finding container e224f604d7c255c6fb131273e9945403bfe5be95e51fa5abcc93306346bf69ea: Status 404 returned error can't find the container with id e224f604d7c255c6fb131273e9945403bfe5be95e51fa5abcc93306346bf69ea Nov 01 22:36:32.703674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:32.703649 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd"] Nov 01 22:36:33.209848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.207516 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" event={"ID":"078d57ed-1850-4485-b27f-b38514f5325a","Type":"ContainerStarted","Data":"284137734d0fd6866ecc4bc82b34b4433a4ef0020bd673e1a8165a5633c0b6f2"} Nov 01 22:36:33.209848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.207586 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" event={"ID":"078d57ed-1850-4485-b27f-b38514f5325a","Type":"ContainerStarted","Data":"e224f604d7c255c6fb131273e9945403bfe5be95e51fa5abcc93306346bf69ea"} Nov 01 22:36:33.230775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.227421 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" podStartSLOduration=2.22740442 podStartE2EDuration="2.22740442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:33.224633083 +0000 UTC m=+805.873132319" watchObservedRunningTime="2026-11-01 22:36:33.22740442 +0000 UTC m=+805.875903668" Nov 01 22:36:33.509424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.509389 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:33.511407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.511389 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:33.563828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.563795 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 22:36:33.564014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.563860 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 22:36:33.608777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.608735 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-xks-ca\") pod \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " Nov 01 22:36:33.609308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.609210 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-xks-ca\") pod \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " Nov 01 22:36:33.609308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.609260 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-kube-api-access-h8v9j\" (UniqueName: \"kubernetes.io/projected/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-kube-api-access-h8v9j\") pod \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " Nov 01 22:36:33.609454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.609424 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-openshift-ca\") pod \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " Nov 01 22:36:33.609558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.609481 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-openshift-ca\") pod \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\" (UID: \"7c6aadea-133d-46e9-a3ba-3b8b81633a0a\") " Nov 01 22:36:33.610303 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.610188 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-openshift-ca" pod "b3c7006c-c170-4188-a7b5-27bfbbb6c82a" (UID: "b3c7006c-c170-4188-a7b5-27bfbbb6c82a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:33.610437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.610364 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-kube-api-access-b6tcx\" (UniqueName: \"kubernetes.io/projected/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-kube-api-access-b6tcx\") pod \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\" (UID: \"b3c7006c-c170-4188-a7b5-27bfbbb6c82a\") " Nov 01 22:36:33.610607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.610581 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-openshift-ca" pod "7c6aadea-133d-46e9-a3ba-3b8b81633a0a" (UID: "7c6aadea-133d-46e9-a3ba-3b8b81633a0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:33.612734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.612707 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:33.612876 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.612860 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:33.612961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.612740 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-kube-api-access-b6tcx" pod "b3c7006c-c170-4188-a7b5-27bfbbb6c82a" (UID: "b3c7006c-c170-4188-a7b5-27bfbbb6c82a"). InnerVolumeSpecName "kube-api-access-b6tcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:33.621630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.621611 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-kube-api-access-h8v9j" pod "7c6aadea-133d-46e9-a3ba-3b8b81633a0a" (UID: "7c6aadea-133d-46e9-a3ba-3b8b81633a0a"). InnerVolumeSpecName "kube-api-access-h8v9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:33.621821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.621808 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-xks-ca" pod "7c6aadea-133d-46e9-a3ba-3b8b81633a0a" (UID: "7c6aadea-133d-46e9-a3ba-3b8b81633a0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:33.623367 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.623340 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-xks-ca" pod "b3c7006c-c170-4188-a7b5-27bfbbb6c82a" (UID: "b3c7006c-c170-4188-a7b5-27bfbbb6c82a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:33.714778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.714700 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:33.714778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.714728 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:33.714778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.714739 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8v9j\" (UniqueName: \"kubernetes.io/projected/7c6aadea-133d-46e9-a3ba-3b8b81633a0a-kube-api-access-h8v9j\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:33.714778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:33.714748 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6tcx\" (UniqueName: \"kubernetes.io/projected/b3c7006c-c170-4188-a7b5-27bfbbb6c82a-kube-api-access-b6tcx\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:34.218428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.218396 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" Nov 01 22:36:34.218650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.218395 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v" event={"ID":"7c6aadea-133d-46e9-a3ba-3b8b81633a0a","Type":"ContainerDied","Data":"b7b5bd75fd0e2cb7a61d995f76af230c891c582d8d405fa42fd77dc26f7123df"} Nov 01 22:36:34.218650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.218580 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7b5bd75fd0e2cb7a61d995f76af230c891c582d8d405fa42fd77dc26f7123df" Nov 01 22:36:34.220194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.220164 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" event={"ID":"b3c7006c-c170-4188-a7b5-27bfbbb6c82a","Type":"ContainerDied","Data":"dd7cfb014dd61dbaa25be8922dea70b6f67b5a45fd427cb8da3edf8c3ace6abe"} Nov 01 22:36:34.220305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.220219 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd7cfb014dd61dbaa25be8922dea70b6f67b5a45fd427cb8da3edf8c3ace6abe" Nov 01 22:36:34.220305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.220179 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg" Nov 01 22:36:34.222219 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.222192 6287 generic.go:356] "Generic (PLEG): container finished" podID="078d57ed-1850-4485-b27f-b38514f5325a" containerID="284137734d0fd6866ecc4bc82b34b4433a4ef0020bd673e1a8165a5633c0b6f2" exitCode=0 Nov 01 22:36:34.222336 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.222263 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" event={"ID":"078d57ed-1850-4485-b27f-b38514f5325a","Type":"ContainerDied","Data":"284137734d0fd6866ecc4bc82b34b4433a4ef0020bd673e1a8165a5633c0b6f2"} Nov 01 22:36:34.564397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.564363 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 22:36:34.564552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:34.564433 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 22:36:35.563845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.563754 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 22:36:35.563845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.563826 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 22:36:35.714033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.713740 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:35.835966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.835860 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-openshift-ca\") pod \"078d57ed-1850-4485-b27f-b38514f5325a\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " Nov 01 22:36:35.835966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.835910 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-xks-ca\") pod \"078d57ed-1850-4485-b27f-b38514f5325a\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " Nov 01 22:36:35.835966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.835949 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078d57ed-1850-4485-b27f-b38514f5325a-kube-api-access-l5gbz\" (UniqueName: \"kubernetes.io/projected/078d57ed-1850-4485-b27f-b38514f5325a-kube-api-access-l5gbz\") pod \"078d57ed-1850-4485-b27f-b38514f5325a\" (UID: \"078d57ed-1850-4485-b27f-b38514f5325a\") " Nov 01 22:36:35.836573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.836541 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-openshift-ca" pod "078d57ed-1850-4485-b27f-b38514f5325a" (UID: "078d57ed-1850-4485-b27f-b38514f5325a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:35.838304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.838274 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-xks-ca" pod "078d57ed-1850-4485-b27f-b38514f5325a" (UID: "078d57ed-1850-4485-b27f-b38514f5325a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:35.838390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.838303 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078d57ed-1850-4485-b27f-b38514f5325a-kube-api-access-l5gbz" pod "078d57ed-1850-4485-b27f-b38514f5325a" (UID: "078d57ed-1850-4485-b27f-b38514f5325a"). InnerVolumeSpecName "kube-api-access-l5gbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:35.938498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.938454 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:35.938646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.938507 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078d57ed-1850-4485-b27f-b38514f5325a-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:35.938646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:35.938524 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5gbz\" (UniqueName: \"kubernetes.io/projected/078d57ed-1850-4485-b27f-b38514f5325a-kube-api-access-l5gbz\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:36.329183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:36.325035 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" Nov 01 22:36:36.329183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:36.325058 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd" event={"ID":"078d57ed-1850-4485-b27f-b38514f5325a","Type":"ContainerDied","Data":"e224f604d7c255c6fb131273e9945403bfe5be95e51fa5abcc93306346bf69ea"} Nov 01 22:36:36.329183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:36.325106 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e224f604d7c255c6fb131273e9945403bfe5be95e51fa5abcc93306346bf69ea" Nov 01 22:36:36.564244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:36.564212 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 01 22:36:36.564704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:36.564269 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 01 22:36:38.424962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.424819 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:38.685259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.626004 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:38.685259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.626069 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:38.691783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.691749 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:38.693698 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.693664 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:38.722839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.722798 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:38.809817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.809785 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk"] Nov 01 22:36:38.830303 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.819400 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b3c7006c-c170-4188-a7b5-27bfbbb6c82a" containerName="revoke-keys" Nov 01 22:36:38.830578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.830550 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c7006c-c170-4188-a7b5-27bfbbb6c82a" containerName="revoke-keys" Nov 01 22:36:38.830732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.830718 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7c6aadea-133d-46e9-a3ba-3b8b81633a0a" containerName="revoke-keys" Nov 01 22:36:38.830812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.830802 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6aadea-133d-46e9-a3ba-3b8b81633a0a" containerName="revoke-keys" Nov 01 22:36:38.830971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.830960 6287 cpu_manager.go:414] "containerMap: removing container" podUID="078d57ed-1850-4485-b27f-b38514f5325a" containerName="revoke-keys" Nov 01 22:36:38.831055 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.831044 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078d57ed-1850-4485-b27f-b38514f5325a" containerName="revoke-keys" Nov 01 22:36:38.831423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.831404 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="078d57ed-1850-4485-b27f-b38514f5325a" containerName="revoke-keys" Nov 01 22:36:38.831545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.831534 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c7006c-c170-4188-a7b5-27bfbbb6c82a" containerName="revoke-keys" Nov 01 22:36:38.850939 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.850901 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6aadea-133d-46e9-a3ba-3b8b81633a0a" containerName="revoke-keys" Nov 01 22:36:38.901153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.866397 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.901153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.868728 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:38.901153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.871075 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:38.903103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.903059 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.915390 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-certs\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.915437 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-socket\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.915478 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-credential-socket\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.915583 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044f3498-9c73-4231-9035-449af1735d27-istio-podinfo\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.915641 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044f3498-9c73-4231-9035-449af1735d27-istiod-ca-cert\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.915673 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgl8k\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-kube-api-access-zgl8k\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.915781 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-envoy\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.915832 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-istio-token\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.915903 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-data\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.916098 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.925962 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.948048 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-26aaf165-openshift-default-dockercfg-7r599\"" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.948638 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.953314 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:38.988189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:38.962191 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.027357 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.029809 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-certs\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.029852 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-socket\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.029889 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-credential-socket\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.029986 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044f3498-9c73-4231-9035-449af1735d27-istio-podinfo\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.030037 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044f3498-9c73-4231-9035-449af1735d27-istiod-ca-cert\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.030065 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgl8k\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-kube-api-access-zgl8k\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.030185 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-envoy\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.030239 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-istio-token\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.030308 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-data\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.030849 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-data\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.043439 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044f3498-9c73-4231-9035-449af1735d27-istiod-ca-cert\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.043734 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-socket\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.050890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.043995 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-credential-socket\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.101970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.069430 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:36:39.101970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.091872 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-certs\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.172136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.147534 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgl8k\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-kube-api-access-zgl8k\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.203136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.191758 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-istio-token\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.237138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.214975 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044f3498-9c73-4231-9035-449af1735d27-istio-podinfo\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.268938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.268233 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.372411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.353232 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.432136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.431229 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:39.432136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.432020 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:36:39.492138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.458192 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk"] Nov 01 22:36:39.554436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.539595 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.670383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.575625 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-envoy\") pod \"e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.670383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.641502 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:39.840221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.759317 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:39.840221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.759399 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.840221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.779691 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:36:39.942167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.916500 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.942167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.916885 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.942167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.918296 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.950142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.942700 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:39.956149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.951660 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.956541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.956510 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:39.962886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:39.962844 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.060689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.060658 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:40.099234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.073104 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.327569 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.419547 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.426825 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.426908 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.426945 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.426983 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.427016 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.427049 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.427084 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.427173 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.427210 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.427242 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.427274 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.427309 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.478283 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.479555 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.540677 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.718216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.541264 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:36:40.794264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.742936 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:40.794264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.743014 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:40.930150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.929375 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:40.930150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.929451 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:41.003774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:40.988583 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:36:41.157811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.108047 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:41.157811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.116642 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:41.248828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.248637 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:41.405138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.401146 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:41.837550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.465302 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:41.837550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.612380 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:36:41.837550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.728565 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:41.837550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.777251 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:36:41.837550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.777310 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:41.837550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.777334 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:36:41.861148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.777368 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:41.861148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.777391 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Nov 01 22:36:41.861148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.777411 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:41.861148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.777441 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:36:41.861148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.777462 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:41.861148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.790627 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:36:41.861148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.790680 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 01 22:36:41.887138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.864783 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:41.887419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:41.887378 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.068733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.042758 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded" Nov 01 22:36:42.110679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.110237 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.254267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.254214 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.255250 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.251583 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.572490 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.572450 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:42.604288 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.604244 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.780236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.780195 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:42.780479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.780450 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.852203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.825242 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:36:42.852203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.825644 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:36:42.852203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.825946 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.852203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.826239 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:42.852203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.826497 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:36:42.852203 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:42.832264 6287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.12s" Nov 01 22:36:42.852203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.832298 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp"] Nov 01 22:36:42.913171 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.865972 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.964911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.870712 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:42.967416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.870744 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:42.968428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.870766 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:36:42.970150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.870806 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:36:42.986269 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.870823 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:42.986786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.925847 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:42.987010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:42.986976 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:43.138799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.138706 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:43.278565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.264245 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:43.321149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.297670 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp"] Nov 01 22:36:43.337180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.321585 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:36:43.395196 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:43.340087 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:43.728275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.299263 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:43.888136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.371334 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpk96\" (UniqueName: \"kubernetes.io/projected/dbbbd768-2180-4d65-80a0-e3a7c228d417-kube-api-access-hpk96\") pod \"maas-api-revoke-keys-e2e-ait-fd214936-wf5rp\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:43.888136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.825787 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd214936-wf5rp\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:43.888136 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:43.605739 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:43.888136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.852737 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:36:43.888136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.852827 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:36:43.888136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.852875 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:36:43.888136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.852956 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:36:43.888136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.420208 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:36:43.920266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.920172 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd214936-wf5rp\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:44.334597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.927225 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:36:44.335196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.927261 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:36:44.411440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.411383 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.411743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.927307 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.412235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.927325 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:36:44.413566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.016259 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.414397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.016307 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.419174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.073730 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:44.424189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.073770 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:44.429177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.073789 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:44.434773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.073809 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:36:44.443194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.281427 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:44.453206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.288198 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.525632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:43.927281 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:36:44.559153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.559100 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd214936-wf5rp\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:44.559573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.559546 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpk96\" (UniqueName: \"kubernetes.io/projected/dbbbd768-2180-4d65-80a0-e3a7c228d417-kube-api-access-hpk96\") pod \"maas-api-revoke-keys-e2e-ait-fd214936-wf5rp\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:44.563445 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.563411 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd214936-wf5rp\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:44.591222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.582281 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd214936-wf5rp\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:44.635060 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.635008 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.792153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.731434 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.792153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.731496 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:44.792153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.742386 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:44.792153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.742444 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:36:44.900186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:44.878413 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:44.902159 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:44.881362 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:44.937906 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:44.896033 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:45.181320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.181217 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.181907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.181869 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.205794 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.205759 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd214936-wf5rp\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:45.430142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.402935 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpk96\" (UniqueName: \"kubernetes.io/projected/dbbbd768-2180-4d65-80a0-e3a7c228d417-kube-api-access-hpk96\") pod \"maas-api-revoke-keys-e2e-ait-fd214936-wf5rp\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:45.577215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.430483 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.577812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.432151 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:45.578659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.432210 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:36:45.579088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.432233 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:36:45.579839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.432259 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:45.580286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.432294 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:36:45.580980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.432316 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:45.581417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.433222 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.582651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.480656 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.584447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.497160 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.584832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.502312 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:36:45.585212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.572058 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.602741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.602699 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:36:45.669140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.669065 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:45.998321 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.998278 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:45.998931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.998891 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.999306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.999281 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:45.999459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.999438 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:45.999768 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:45.999745 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.407140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.404774 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" event={"ID":"044f3498-9c73-4231-9035-449af1735d27","Type":"ContainerStarted","Data":"a1f60e32b874c4331f778ba78b80d75bf3f08becba6e8ee79ec71cf2006c6483"} Nov 01 22:36:46.585398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.585352 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.586013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.585980 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:46.586516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.586489 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:46.586938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.586904 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:36:46.587370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.587346 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:36:46.587783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.587759 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:46.588191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.588166 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.588593 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.588564 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:46.588990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.588966 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:36:46.589437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.589412 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.589839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.589816 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:46.590252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.590228 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:46.607817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:46.607769 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:47.081138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.041408 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:47.081138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.056213 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded" Nov 01 22:36:47.103449 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:47.103408 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:36:47.350153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.342071 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:36:47.436287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.436241 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:47.613095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.610453 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:36:47.613095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.610793 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:36:47.613095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.611030 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:36:47.613095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.611270 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:36:47.613095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.611650 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:36:47.613095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.611880 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:36:47.613095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.612098 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:36:47.614146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.613835 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:36:47.615836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.615813 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8m4jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:36:47.616011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.615993 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" podUID="e65c3aec-3fe4-4c4f-8689-cbabab25e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:47.634037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.633999 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:47.700164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.699170 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:36:47.733650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.733589 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:47.739030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:47.738993 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:48.075342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.075301 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:36:48.180172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.177495 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:36:48.180172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.177571 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:36:48.180172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.177591 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" event={"ID":"044f3498-9c73-4231-9035-449af1735d27","Type":"ContainerStarted","Data":"bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e"} Nov 01 22:36:48.302869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.258614 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:36:48.534133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.533830 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:36:48.582751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.582665 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:36:48.642236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.641036 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:36:48.642236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.641101 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:36:48.711886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.645419 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:48.711886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.645719 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:36:48.711886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.662204 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:36:48.711886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.674550 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:36:48.715446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.715409 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:36:48.736150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.732834 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:48.778136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.763779 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:36:48.779771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.771515 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:36:48.785696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.785665 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:36:48.787685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.787658 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:36:48.789605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.789571 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:48.857235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.857157 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk"] Nov 01 22:36:48.895441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.893703 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:36:48.907940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.907884 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" podStartSLOduration=10.907865307 podStartE2EDuration="10.907865307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:48.192513492 +0000 UTC m=+820.841012727" watchObservedRunningTime="2026-11-01 22:36:48.907865307 +0000 UTC m=+821.556364541" Nov 01 22:36:48.911987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.911960 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp"] Nov 01 22:36:48.952164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.952132 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" event={"ID":"dbbbd768-2180-4d65-80a0-e3a7c228d417","Type":"ContainerStarted","Data":"103696465f32c978b80c48448827f23f37a5577eda90aeecd52f7e341d31b24c"} Nov 01 22:36:48.952361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.952187 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:48.952361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.952211 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" event={"ID":"dbbbd768-2180-4d65-80a0-e3a7c228d417","Type":"ContainerStarted","Data":"8c1ae8e0eaf868473cb89c8c6766b92a7915785a6ffe0d28e6b20d6889036e86"} Nov 01 22:36:48.956197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:48.956126 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:36:49.065888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:49.065408 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" podStartSLOduration=11.065388035 podStartE2EDuration="11.065388035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:49.033971155 +0000 UTC m=+821.682470415" watchObservedRunningTime="2026-11-01 22:36:49.065388035 +0000 UTC m=+821.713887272" Nov 01 22:36:49.963926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:49.963890 6287 generic.go:356] "Generic (PLEG): container finished" podID="dbbbd768-2180-4d65-80a0-e3a7c228d417" containerID="103696465f32c978b80c48448827f23f37a5577eda90aeecd52f7e341d31b24c" exitCode=0 Nov 01 22:36:49.964346 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:49.963975 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" event={"ID":"dbbbd768-2180-4d65-80a0-e3a7c228d417","Type":"ContainerDied","Data":"103696465f32c978b80c48448827f23f37a5577eda90aeecd52f7e341d31b24c"} Nov 01 22:36:51.198938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.198906 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:51.211898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.211839 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbbd768-2180-4d65-80a0-e3a7c228d417-kube-api-access-hpk96\" (UniqueName: \"kubernetes.io/projected/dbbbd768-2180-4d65-80a0-e3a7c228d417-kube-api-access-hpk96\") pod \"dbbbd768-2180-4d65-80a0-e3a7c228d417\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " Nov 01 22:36:51.212152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.211945 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-xks-ca\") pod \"dbbbd768-2180-4d65-80a0-e3a7c228d417\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " Nov 01 22:36:51.215719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.215685 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-xks-ca" pod "dbbbd768-2180-4d65-80a0-e3a7c228d417" (UID: "dbbbd768-2180-4d65-80a0-e3a7c228d417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:51.216432 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.216409 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbbd768-2180-4d65-80a0-e3a7c228d417-kube-api-access-hpk96" pod "dbbbd768-2180-4d65-80a0-e3a7c228d417" (UID: "dbbbd768-2180-4d65-80a0-e3a7c228d417"). InnerVolumeSpecName "kube-api-access-hpk96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:51.313768 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.313506 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-openshift-ca\") pod \"dbbbd768-2180-4d65-80a0-e3a7c228d417\" (UID: \"dbbbd768-2180-4d65-80a0-e3a7c228d417\") " Nov 01 22:36:51.314010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.313968 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-openshift-ca" pod "dbbbd768-2180-4d65-80a0-e3a7c228d417" (UID: "dbbbd768-2180-4d65-80a0-e3a7c228d417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:51.314404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.314360 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpk96\" (UniqueName: \"kubernetes.io/projected/dbbbd768-2180-4d65-80a0-e3a7c228d417-kube-api-access-hpk96\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:51.314404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.314395 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:51.314667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.314412 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbbbd768-2180-4d65-80a0-e3a7c228d417-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:51.502146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.502093 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k"] Nov 01 22:36:51.505192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.505163 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dbbbd768-2180-4d65-80a0-e3a7c228d417" containerName="revoke-keys" Nov 01 22:36:51.505192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.505190 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbbd768-2180-4d65-80a0-e3a7c228d417" containerName="revoke-keys" Nov 01 22:36:51.506398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.505459 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbbd768-2180-4d65-80a0-e3a7c228d417" containerName="revoke-keys" Nov 01 22:36:51.517023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.516990 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k"] Nov 01 22:36:51.517197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.517161 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:36:51.524424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.524386 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-26aaf165\"" Nov 01 22:36:51.524705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.524681 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-26aaf165-dockercfg-k97cr\"" Nov 01 22:36:51.649135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.648066 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7jd\" (UniqueName: \"kubernetes.io/projected/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-kube-api-access-lh7jd\") pod \"payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k\" (UID: \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\") " pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:36:51.653784 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.653250 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-plugins-config-volume\") pod \"payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k\" (UID: \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\") " pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:36:51.767186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.755720 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7jd\" (UniqueName: \"kubernetes.io/projected/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-kube-api-access-lh7jd\") pod \"payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k\" (UID: \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\") " pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:36:51.767186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.755824 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-plugins-config-volume\") pod \"payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k\" (UID: \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\") " pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:36:51.767186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.757084 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-plugins-config-volume\") pod \"payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k\" (UID: \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\") " pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:36:51.783361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.775876 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7jd\" (UniqueName: \"kubernetes.io/projected/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-kube-api-access-lh7jd\") pod \"payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k\" (UID: \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\") " pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:36:51.839134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:51.838409 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:36:52.013633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:52.013601 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" Nov 01 22:36:52.014058 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:52.013697 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp" event={"ID":"dbbbd768-2180-4d65-80a0-e3a7c228d417","Type":"ContainerDied","Data":"8c1ae8e0eaf868473cb89c8c6766b92a7915785a6ffe0d28e6b20d6889036e86"} Nov 01 22:36:52.014239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:52.014076 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c1ae8e0eaf868473cb89c8c6766b92a7915785a6ffe0d28e6b20d6889036e86" Nov 01 22:36:52.166942 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:36:52.166913 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d951c5b_73cc_4f5d_a0ed_87085e0cd05d.slice/crio-543097cb191748c6c8077ec4c01d328183549897d5f94304a20890a36e723ed2 WatchSource:0}: Error finding container 543097cb191748c6c8077ec4c01d328183549897d5f94304a20890a36e723ed2: Status 404 returned error can't find the container with id 543097cb191748c6c8077ec4c01d328183549897d5f94304a20890a36e723ed2 Nov 01 22:36:52.173360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:52.173336 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k"] Nov 01 22:36:53.025357 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:53.025319 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" event={"ID":"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d","Type":"ContainerStarted","Data":"9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f"} Nov 01 22:36:53.025753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:53.025416 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" event={"ID":"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d","Type":"ContainerStarted","Data":"543097cb191748c6c8077ec4c01d328183549897d5f94304a20890a36e723ed2"} Nov 01 22:36:53.025753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:53.025447 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:36:53.055857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:53.055808 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" podStartSLOduration=2.055794159 podStartE2EDuration="2.055794159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:53.048612354 +0000 UTC m=+825.697111588" watchObservedRunningTime="2026-11-01 22:36:53.055794159 +0000 UTC m=+825.704293393" Nov 01 22:36:53.973312 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:53.973276 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:54.313934 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:36:54.310955 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:36:57.791205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:57.791160 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v"] Nov 01 22:36:57.799756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:57.799724 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0500c01f-plx5v"] Nov 01 22:36:59.187244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.187184 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6"] Nov 01 22:36:59.359616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.359579 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc"] Nov 01 22:36:59.364440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.364416 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.376869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.376842 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-61ce6e07-gw-openshift-default-dockercfg-k6tsp\"" Nov 01 22:36:59.383790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.383761 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc"] Nov 01 22:36:59.473602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.473526 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6aadea-133d-46e9-a3ba-3b8b81633a0a" path="/var/lib/kubelet/pods/7c6aadea-133d-46e9-a3ba-3b8b81633a0a/volumes" Nov 01 22:36:59.477237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.477215 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzfn5\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-kube-api-access-dzfn5\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.477323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.477259 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842d9100-2f7b-458d-a262-c77722176e60-istio-podinfo\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.477396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.477364 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-envoy\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.477489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.477472 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-data\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.477538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.477515 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842d9100-2f7b-458d-a262-c77722176e60-istiod-ca-cert\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.477590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.477546 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-credential-socket\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.477590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.477563 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-socket\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.477694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.477679 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-istio-token\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.477747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.477713 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-certs\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579007 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.578974 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-istio-token\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579025 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-certs\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579291 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579227 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzfn5\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-kube-api-access-dzfn5\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579291 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579281 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842d9100-2f7b-458d-a262-c77722176e60-istio-podinfo\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579368 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-envoy\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579466 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-data\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579479 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-certs\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579663 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842d9100-2f7b-458d-a262-c77722176e60-istiod-ca-cert\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579715 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-credential-socket\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579738 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-socket\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.579924 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.579861 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-data\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.580222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.580180 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-credential-socket\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.580317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.580196 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-socket\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.580407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.580386 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842d9100-2f7b-458d-a262-c77722176e60-istiod-ca-cert\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.581950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.581924 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-envoy\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.582036 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.581990 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842d9100-2f7b-458d-a262-c77722176e60-istio-podinfo\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.591194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.591169 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-istio-token\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.591583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.591550 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzfn5\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-kube-api-access-dzfn5\") pod \"e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:36:59.683316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:36:59.683267 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:37:00.160739 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:37:00.160691 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842d9100_2f7b_458d_a262_c77722176e60.slice/crio-b1620a69e618398c38722bd405909462b956fa0ce4813065788a5c4b7108ec7b WatchSource:0}: Error finding container b1620a69e618398c38722bd405909462b956fa0ce4813065788a5c4b7108ec7b: Status 404 returned error can't find the container with id b1620a69e618398c38722bd405909462b956fa0ce4813065788a5c4b7108ec7b Nov 01 22:37:00.163965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:00.163928 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:37:00.164066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:00.164001 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:37:00.164066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:00.164031 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:37:00.165043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:00.165019 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc"] Nov 01 22:37:01.150634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:01.150597 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" event={"ID":"842d9100-2f7b-458d-a262-c77722176e60","Type":"ContainerStarted","Data":"1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925"} Nov 01 22:37:01.151036 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:01.150690 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" event={"ID":"842d9100-2f7b-458d-a262-c77722176e60","Type":"ContainerStarted","Data":"b1620a69e618398c38722bd405909462b956fa0ce4813065788a5c4b7108ec7b"} Nov 01 22:37:01.184347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:01.184297 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" podStartSLOduration=2.184282422 podStartE2EDuration="2.184282422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:01.173334991 +0000 UTC m=+833.821834226" watchObservedRunningTime="2026-11-01 22:37:01.184282422 +0000 UTC m=+833.832781657" Nov 01 22:37:01.684626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:01.684593 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:37:01.690174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:01.690147 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:37:02.159715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:02.159684 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:37:02.160723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:02.160702 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:37:04.037416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:04.037391 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:37:04.057320 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:37:04.057289 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e4b2bc_b479_44cd_9532_f3ef6ae75452.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:37:04.188173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:04.188091 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" podUID="7d4ca24e-dbb4-4d92-a60f-412aae7f8e71" containerName="payload-processing" containerID="cri-o://f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514" gracePeriod=25 Nov 01 22:37:04.480651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:04.480628 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:37:04.585754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:04.585660 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-plugins-config-volume\") pod \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\" (UID: \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\") " Nov 01 22:37:04.585898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:04.585865 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-kube-api-access-w2t4r\" (UniqueName: \"kubernetes.io/projected/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-kube-api-access-w2t4r\") pod \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\" (UID: \"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71\") " Nov 01 22:37:04.586101 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:04.586072 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-plugins-config-volume" pod "7d4ca24e-dbb4-4d92-a60f-412aae7f8e71" (UID: "7d4ca24e-dbb4-4d92-a60f-412aae7f8e71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:04.586689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:04.586658 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:04.589465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:04.589439 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-kube-api-access-w2t4r" pod "7d4ca24e-dbb4-4d92-a60f-412aae7f8e71" (UID: "7d4ca24e-dbb4-4d92-a60f-412aae7f8e71"). InnerVolumeSpecName "kube-api-access-w2t4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:04.709771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:04.709595 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2t4r\" (UniqueName: \"kubernetes.io/projected/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71-kube-api-access-w2t4r\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:05.194631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.194597 6287 generic.go:356] "Generic (PLEG): container finished" podID="7d4ca24e-dbb4-4d92-a60f-412aae7f8e71" containerID="f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514" exitCode=0 Nov 01 22:37:05.194631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.194643 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" Nov 01 22:37:05.195194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.194654 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" event={"ID":"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71","Type":"ContainerDied","Data":"f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514"} Nov 01 22:37:05.195194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.194706 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6" event={"ID":"7d4ca24e-dbb4-4d92-a60f-412aae7f8e71","Type":"ContainerDied","Data":"f0139910778fd5cb123e1b950a773226b49c31a895bec4f4b4940e9360d10179"} Nov 01 22:37:05.195194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.194741 6287 scope.go:122] "RemoveContainer" containerID="f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514" Nov 01 22:37:05.206528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.206508 6287 scope.go:122] "RemoveContainer" containerID="f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514" Nov 01 22:37:05.206779 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:37:05.206749 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514\": container with ID starting with f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514 not found: ID does not exist" containerID="f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514" Nov 01 22:37:05.206885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.206788 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514"} err="failed to get container status \"f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514\": rpc error: code = NotFound desc = could not find container \"f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514\": container with ID starting with f8a808754f0cd73d235ea3ab0d93e0779451a604b0d05ddfa44e98ee98843514 not found: ID does not exist" Nov 01 22:37:05.222443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.222414 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6"] Nov 01 22:37:05.228946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.228927 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd214936-9f5f55f9c-dgkg6"] Nov 01 22:37:05.474339 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:05.474258 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4ca24e-dbb4-4d92-a60f-412aae7f8e71" path="/var/lib/kubelet/pods/7d4ca24e-dbb4-4d92-a60f-412aae7f8e71/volumes" Nov 01 22:37:06.216609 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:06.216571 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp"] Nov 01 22:37:06.225616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:06.225586 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd214936-wf5rp"] Nov 01 22:37:07.474898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:07.474867 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbbbd768-2180-4d65-80a0-e3a7c228d417" path="/var/lib/kubelet/pods/dbbbd768-2180-4d65-80a0-e3a7c228d417/volumes" Nov 01 22:37:09.757898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.757858 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs"] Nov 01 22:37:09.758790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.758766 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7d4ca24e-dbb4-4d92-a60f-412aae7f8e71" containerName="payload-processing" Nov 01 22:37:09.758909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.758791 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4ca24e-dbb4-4d92-a60f-412aae7f8e71" containerName="payload-processing" Nov 01 22:37:09.759045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.759030 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4ca24e-dbb4-4d92-a60f-412aae7f8e71" containerName="payload-processing" Nov 01 22:37:09.764951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.764926 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:09.771468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.771433 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs"] Nov 01 22:37:09.771880 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.771852 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:37:09.771995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.771852 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:09.771995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.771852 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:37:09.771995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.771932 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:37:09.822306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.822261 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:09.822683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.822653 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56k9\" (UniqueName: \"kubernetes.io/projected/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-kube-api-access-g56k9\") pod \"maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:09.822810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.822799 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:09.924820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.924787 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56k9\" (UniqueName: \"kubernetes.io/projected/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-kube-api-access-g56k9\") pod \"maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:09.924971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.924873 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:09.925026 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.925004 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:09.925588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.925557 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:09.927254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.927237 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:09.933174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:09.933154 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56k9\" (UniqueName: \"kubernetes.io/projected/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-kube-api-access-g56k9\") pod \"maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:10.084499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:10.084418 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:10.444975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:10.444941 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs"] Nov 01 22:37:11.318699 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:11.318658 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" event={"ID":"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e","Type":"ContainerStarted","Data":"fea52b816f6646bd672d84ac616b37982cf3ee500c87d68a8b4608788aa5f2bb"} Nov 01 22:37:11.319237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:11.318724 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" event={"ID":"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e","Type":"ContainerStarted","Data":"94607c5387da8d018ef292b5129ce7fb1ba66547e903bfdddac27d64a362b193"} Nov 01 22:37:11.339610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:11.339560 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" podStartSLOduration=2.339546357 podStartE2EDuration="2.339546357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:11.334909235 +0000 UTC m=+843.983408472" watchObservedRunningTime="2026-11-01 22:37:11.339546357 +0000 UTC m=+843.988045592" Nov 01 22:37:12.377137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:12.376412 6287 generic.go:356] "Generic (PLEG): container finished" podID="a3a3fa5c-3610-4f02-8e27-be24a4b31b1e" containerID="fea52b816f6646bd672d84ac616b37982cf3ee500c87d68a8b4608788aa5f2bb" exitCode=0 Nov 01 22:37:12.377137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:12.376604 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" event={"ID":"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e","Type":"ContainerDied","Data":"fea52b816f6646bd672d84ac616b37982cf3ee500c87d68a8b4608788aa5f2bb"} Nov 01 22:37:13.827425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.827395 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:13.870664 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.870628 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-xks-ca\") pod \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " Nov 01 22:37:13.870867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.870792 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-kube-api-access-g56k9\" (UniqueName: \"kubernetes.io/projected/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-kube-api-access-g56k9\") pod \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " Nov 01 22:37:13.870926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.870872 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-openshift-ca\") pod \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\" (UID: \"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e\") " Nov 01 22:37:13.872262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.872231 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-openshift-ca" pod "a3a3fa5c-3610-4f02-8e27-be24a4b31b1e" (UID: "a3a3fa5c-3610-4f02-8e27-be24a4b31b1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:13.885239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.882289 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-xks-ca" pod "a3a3fa5c-3610-4f02-8e27-be24a4b31b1e" (UID: "a3a3fa5c-3610-4f02-8e27-be24a4b31b1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:13.885239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.882329 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-kube-api-access-g56k9" pod "a3a3fa5c-3610-4f02-8e27-be24a4b31b1e" (UID: "a3a3fa5c-3610-4f02-8e27-be24a4b31b1e"). InnerVolumeSpecName "kube-api-access-g56k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:13.975210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.975157 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.975210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.975206 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:13.975210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:13.975222 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56k9\" (UniqueName: \"kubernetes.io/projected/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e-kube-api-access-g56k9\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:14.486467 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:14.486406 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" event={"ID":"a3a3fa5c-3610-4f02-8e27-be24a4b31b1e","Type":"ContainerDied","Data":"94607c5387da8d018ef292b5129ce7fb1ba66547e903bfdddac27d64a362b193"} Nov 01 22:37:14.486467 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:14.486472 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94607c5387da8d018ef292b5129ce7fb1ba66547e903bfdddac27d64a362b193" Nov 01 22:37:14.486759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:14.486536 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs" Nov 01 22:37:15.762137 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:37:15.751565 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:37:15.830234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:15.830191 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:15.856229 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:15.856183 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:15.865259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:15.865213 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:15.872769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:15.872734 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:15.888265 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:15.888229 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:15.997242 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:15.997195 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:16.252244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:16.252195 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:16.731532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:16.731434 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:37:16.831771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:16.831721 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:16.832918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:16.832891 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:37:16.876538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:16.876492 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:16.877297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:16.877273 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:37:16.891169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:16.890383 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:16.891169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:16.890927 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:37:17.239629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:17.236473 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:37:17.328300 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:17.328270 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:37:17.370053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:17.369979 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:37:19.898402 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:19.898362 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k"] Nov 01 22:37:24.898802 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:24.898732 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" podUID="1d951c5b-73cc-4f5d-a0ed-87085e0cd05d" containerName="payload-processing" containerID="cri-o://9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f" gracePeriod=25 Nov 01 22:37:25.268996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.268962 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:37:25.334724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.334687 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-kube-api-access-lh7jd\" (UniqueName: \"kubernetes.io/projected/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-kube-api-access-lh7jd\") pod \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\" (UID: \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\") " Nov 01 22:37:25.335124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.335048 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-plugins-config-volume\") pod \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\" (UID: \"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d\") " Nov 01 22:37:25.337054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.336964 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-plugins-config-volume" pod "1d951c5b-73cc-4f5d-a0ed-87085e0cd05d" (UID: "1d951c5b-73cc-4f5d-a0ed-87085e0cd05d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:25.340706 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.340674 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-kube-api-access-lh7jd" pod "1d951c5b-73cc-4f5d-a0ed-87085e0cd05d" (UID: "1d951c5b-73cc-4f5d-a0ed-87085e0cd05d"). InnerVolumeSpecName "kube-api-access-lh7jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:25.438037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.438004 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7jd\" (UniqueName: \"kubernetes.io/projected/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-kube-api-access-lh7jd\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:25.438037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.438034 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:25.906103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.906008 6287 generic.go:356] "Generic (PLEG): container finished" podID="1d951c5b-73cc-4f5d-a0ed-87085e0cd05d" containerID="9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f" exitCode=0 Nov 01 22:37:25.906103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.906063 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" Nov 01 22:37:25.906569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.906090 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" event={"ID":"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d","Type":"ContainerDied","Data":"9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f"} Nov 01 22:37:25.906569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.906183 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k" event={"ID":"1d951c5b-73cc-4f5d-a0ed-87085e0cd05d","Type":"ContainerDied","Data":"543097cb191748c6c8077ec4c01d328183549897d5f94304a20890a36e723ed2"} Nov 01 22:37:25.906569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.906207 6287 scope.go:122] "RemoveContainer" containerID="9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f" Nov 01 22:37:25.919084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.919062 6287 scope.go:122] "RemoveContainer" containerID="9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f" Nov 01 22:37:25.919388 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:37:25.919368 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f\": container with ID starting with 9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f not found: ID does not exist" containerID="9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f" Nov 01 22:37:25.919449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.919396 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f"} err="failed to get container status \"9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f\": rpc error: code = NotFound desc = could not find container \"9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f\": container with ID starting with 9f1b9d0ccf855e1121dcf6bb4d2cf3e6af094ac5eceba19ea5a39c46a4aa598f not found: ID does not exist" Nov 01 22:37:25.940368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.940335 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k"] Nov 01 22:37:25.947035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:25.947014 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26aaf165-ffb76d7dc-gf44k"] Nov 01 22:37:27.473540 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:27.473510 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d951c5b-73cc-4f5d-a0ed-87085e0cd05d" path="/var/lib/kubelet/pods/1d951c5b-73cc-4f5d-a0ed-87085e0cd05d/volumes" Nov 01 22:37:28.039639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.039604 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2"] Nov 01 22:37:28.040600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.040569 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a3a3fa5c-3610-4f02-8e27-be24a4b31b1e" containerName="revoke-keys" Nov 01 22:37:28.040600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.040600 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a3fa5c-3610-4f02-8e27-be24a4b31b1e" containerName="revoke-keys" Nov 01 22:37:28.040789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.040637 6287 cpu_manager.go:414] "containerMap: removing container" podUID="1d951c5b-73cc-4f5d-a0ed-87085e0cd05d" containerName="payload-processing" Nov 01 22:37:28.040789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.040648 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d951c5b-73cc-4f5d-a0ed-87085e0cd05d" containerName="payload-processing" Nov 01 22:37:28.040904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.040888 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d951c5b-73cc-4f5d-a0ed-87085e0cd05d" containerName="payload-processing" Nov 01 22:37:28.040960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.040921 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a3fa5c-3610-4f02-8e27-be24a4b31b1e" containerName="revoke-keys" Nov 01 22:37:28.052959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.052915 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2"] Nov 01 22:37:28.053203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.053040 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.056650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.056625 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:37:28.057362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.057334 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:28.057680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.057644 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:37:28.057799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.057689 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:37:28.207860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.207827 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.208027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.207889 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5xp\" (UniqueName: \"kubernetes.io/projected/0f2f28e0-5c15-4c55-887f-f808d031924d-kube-api-access-mv5xp\") pod \"maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.208214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.208192 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.309692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.309619 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.309692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.309675 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.309883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.309714 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5xp\" (UniqueName: \"kubernetes.io/projected/0f2f28e0-5c15-4c55-887f-f808d031924d-kube-api-access-mv5xp\") pod \"maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.310464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.310443 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.312170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.312147 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.318532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.318507 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5xp\" (UniqueName: \"kubernetes.io/projected/0f2f28e0-5c15-4c55-887f-f808d031924d-kube-api-access-mv5xp\") pod \"maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.384250 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.384223 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:28.674776 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:37:28.674742 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2f28e0_5c15_4c55_887f_f808d031924d.slice/crio-1c33df78770e156fbb26b4bf9facec25363eeb8db26e18f901f0733868e82d84 WatchSource:0}: Error finding container 1c33df78770e156fbb26b4bf9facec25363eeb8db26e18f901f0733868e82d84: Status 404 returned error can't find the container with id 1c33df78770e156fbb26b4bf9facec25363eeb8db26e18f901f0733868e82d84 Nov 01 22:37:28.677469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.677440 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2"] Nov 01 22:37:28.967168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:28.967008 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" event={"ID":"0f2f28e0-5c15-4c55-887f-f808d031924d","Type":"ContainerStarted","Data":"1c33df78770e156fbb26b4bf9facec25363eeb8db26e18f901f0733868e82d84"} Nov 01 22:37:29.070918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:29.070473 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs"] Nov 01 22:37:29.078957 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:29.078925 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-26aaf165-sbtcs"] Nov 01 22:37:29.203144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:29.203020 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk"] Nov 01 22:37:29.203456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:29.203422 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" podUID="044f3498-9c73-4231-9035-449af1735d27" containerName="istio-proxy" containerID="cri-o://bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e" gracePeriod=30 Nov 01 22:37:29.475518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:29.475487 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a3fa5c-3610-4f02-8e27-be24a4b31b1e" path="/var/lib/kubelet/pods/a3a3fa5c-3610-4f02-8e27-be24a4b31b1e/volumes" Nov 01 22:37:29.978254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:29.978219 6287 generic.go:356] "Generic (PLEG): container finished" podID="0f2f28e0-5c15-4c55-887f-f808d031924d" containerID="d32682551faf630690ef077eab43efefcd74c089671cb7f493700332761946af" exitCode=0 Nov 01 22:37:29.978679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:29.978309 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" event={"ID":"0f2f28e0-5c15-4c55-887f-f808d031924d","Type":"ContainerDied","Data":"d32682551faf630690ef077eab43efefcd74c089671cb7f493700332761946af"} Nov 01 22:37:31.368375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.368350 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:31.562271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.562233 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-xks-ca\") pod \"0f2f28e0-5c15-4c55-887f-f808d031924d\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " Nov 01 22:37:31.562535 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.562422 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2f28e0-5c15-4c55-887f-f808d031924d-kube-api-access-mv5xp\" (UniqueName: \"kubernetes.io/projected/0f2f28e0-5c15-4c55-887f-f808d031924d-kube-api-access-mv5xp\") pod \"0f2f28e0-5c15-4c55-887f-f808d031924d\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " Nov 01 22:37:31.562599 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.562540 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-openshift-ca\") pod \"0f2f28e0-5c15-4c55-887f-f808d031924d\" (UID: \"0f2f28e0-5c15-4c55-887f-f808d031924d\") " Nov 01 22:37:31.563217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.563183 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-openshift-ca" pod "0f2f28e0-5c15-4c55-887f-f808d031924d" (UID: "0f2f28e0-5c15-4c55-887f-f808d031924d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:31.563477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.563451 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:31.565456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.565430 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-xks-ca" pod "0f2f28e0-5c15-4c55-887f-f808d031924d" (UID: "0f2f28e0-5c15-4c55-887f-f808d031924d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:31.569872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.569839 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2f28e0-5c15-4c55-887f-f808d031924d-kube-api-access-mv5xp" pod "0f2f28e0-5c15-4c55-887f-f808d031924d" (UID: "0f2f28e0-5c15-4c55-887f-f808d031924d"). InnerVolumeSpecName "kube-api-access-mv5xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:31.665323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.665289 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2f28e0-5c15-4c55-887f-f808d031924d-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:31.665323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:31.665320 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv5xp\" (UniqueName: \"kubernetes.io/projected/0f2f28e0-5c15-4c55-887f-f808d031924d-kube-api-access-mv5xp\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:32.002934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:32.002903 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" Nov 01 22:37:32.003088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:32.002940 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2" event={"ID":"0f2f28e0-5c15-4c55-887f-f808d031924d","Type":"ContainerDied","Data":"1c33df78770e156fbb26b4bf9facec25363eeb8db26e18f901f0733868e82d84"} Nov 01 22:37:32.003088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:32.002985 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c33df78770e156fbb26b4bf9facec25363eeb8db26e18f901f0733868e82d84" Nov 01 22:37:33.418214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.418181 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9"] Nov 01 22:37:33.418914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.418897 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0f2f28e0-5c15-4c55-887f-f808d031924d" containerName="revoke-keys" Nov 01 22:37:33.418957 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.418914 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2f28e0-5c15-4c55-887f-f808d031924d" containerName="revoke-keys" Nov 01 22:37:33.419064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.419055 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2f28e0-5c15-4c55-887f-f808d031924d" containerName="revoke-keys" Nov 01 22:37:33.423295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.423277 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.440249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.440217 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9"] Nov 01 22:37:33.503224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.503178 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdxgz\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-kube-api-access-cdxgz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.503224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.503225 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.503477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.503337 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.503525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.503473 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.503525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.503507 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.503809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.503790 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.503971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.503829 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.605719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.605679 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.605911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.605734 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.605911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.605869 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdxgz\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-kube-api-access-cdxgz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.605911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.605906 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.606049 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.605949 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.606088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.606054 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.606163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.606093 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.606708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.606677 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.608361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.608335 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.608494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.608475 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.608564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.608548 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.608602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.608561 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.614620 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.614597 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.614814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.614798 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdxgz\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-kube-api-access-cdxgz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9nqs9\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.742065 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.742034 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:33.968160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.968072 6287 patch_prober.go:29] interesting pod/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:37:33.968360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:33.968195 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" podUID="044f3498-9c73-4231-9035-449af1735d27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:37:34.075808 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:37:34.075779 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08dabd9_50ad_49a9_ba37_70a59b58c8c4.slice/crio-9425269bad321aff725101d2110e48241fd44dd7f46939ec4f1ea80e0edce9dd WatchSource:0}: Error finding container 9425269bad321aff725101d2110e48241fd44dd7f46939ec4f1ea80e0edce9dd: Status 404 returned error can't find the container with id 9425269bad321aff725101d2110e48241fd44dd7f46939ec4f1ea80e0edce9dd Nov 01 22:37:34.078388 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.078361 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9"] Nov 01 22:37:34.078622 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.078589 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:37:34.078662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.078652 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:37:34.526889 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.526843 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.630381 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044f3498-9c73-4231-9035-449af1735d27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044f3498-9c73-4231-9035-449af1735d27-istiod-ca-cert\") pod \"044f3498-9c73-4231-9035-449af1735d27\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.630749 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044f3498-9c73-4231-9035-449af1735d27-istiod-ca-cert" pod "044f3498-9c73-4231-9035-449af1735d27" (UID: "044f3498-9c73-4231-9035-449af1735d27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.630904 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-envoy\") pod \"044f3498-9c73-4231-9035-449af1735d27\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.631295 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-certs\") pod \"044f3498-9c73-4231-9035-449af1735d27\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.631588 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-certs" pod "044f3498-9c73-4231-9035-449af1735d27" (UID: "044f3498-9c73-4231-9035-449af1735d27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.631666 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/044f3498-9c73-4231-9035-449af1735d27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044f3498-9c73-4231-9035-449af1735d27-istio-podinfo\") pod \"044f3498-9c73-4231-9035-449af1735d27\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.631982 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-kube-api-access-zgl8k\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-kube-api-access-zgl8k\") pod \"044f3498-9c73-4231-9035-449af1735d27\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.632316 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-data\") pod \"044f3498-9c73-4231-9035-449af1735d27\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.632582 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-data" pod "044f3498-9c73-4231-9035-449af1735d27" (UID: "044f3498-9c73-4231-9035-449af1735d27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.632734 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-socket\") pod \"044f3498-9c73-4231-9035-449af1735d27\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.632962 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-socket" pod "044f3498-9c73-4231-9035-449af1735d27" (UID: "044f3498-9c73-4231-9035-449af1735d27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.633144 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-credential-socket\") pod \"044f3498-9c73-4231-9035-449af1735d27\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.633580 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-credential-socket" pod "044f3498-9c73-4231-9035-449af1735d27" (UID: "044f3498-9c73-4231-9035-449af1735d27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:34.634565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.633693 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-istio-token\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-istio-token\") pod \"044f3498-9c73-4231-9035-449af1735d27\" (UID: \"044f3498-9c73-4231-9035-449af1735d27\") " Nov 01 22:37:34.637005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.634772 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044f3498-9c73-4231-9035-449af1735d27-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:34.637005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.634796 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:34.637005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.634810 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:34.637005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.634826 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:34.637005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.634841 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:34.694133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.689396 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-envoy" pod "044f3498-9c73-4231-9035-449af1735d27" (UID: "044f3498-9c73-4231-9035-449af1735d27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:34.694133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.690217 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-kube-api-access-zgl8k" pod "044f3498-9c73-4231-9035-449af1735d27" (UID: "044f3498-9c73-4231-9035-449af1735d27"). InnerVolumeSpecName "kube-api-access-zgl8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:34.694133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.690574 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-istio-token" pod "044f3498-9c73-4231-9035-449af1735d27" (UID: "044f3498-9c73-4231-9035-449af1735d27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:34.694133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.690732 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/044f3498-9c73-4231-9035-449af1735d27-istio-podinfo" pod "044f3498-9c73-4231-9035-449af1735d27" (UID: "044f3498-9c73-4231-9035-449af1735d27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:37:34.737565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.737524 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:34.737565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.737564 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044f3498-9c73-4231-9035-449af1735d27-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:34.737969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.737581 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044f3498-9c73-4231-9035-449af1735d27-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:34.737969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:34.737598 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgl8k\" (UniqueName: \"kubernetes.io/projected/044f3498-9c73-4231-9035-449af1735d27-kube-api-access-zgl8k\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:35.052782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.052743 6287 generic.go:356] "Generic (PLEG): container finished" podID="044f3498-9c73-4231-9035-449af1735d27" containerID="bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e" exitCode=0 Nov 01 22:37:35.053053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.052830 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" Nov 01 22:37:35.053053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.052825 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" event={"ID":"044f3498-9c73-4231-9035-449af1735d27","Type":"ContainerDied","Data":"bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e"} Nov 01 22:37:35.053053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.053009 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk" event={"ID":"044f3498-9c73-4231-9035-449af1735d27","Type":"ContainerDied","Data":"a1f60e32b874c4331f778ba78b80d75bf3f08becba6e8ee79ec71cf2006c6483"} Nov 01 22:37:35.053053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.053045 6287 scope.go:122] "RemoveContainer" containerID="bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e" Nov 01 22:37:35.055205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.055170 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" event={"ID":"f08dabd9-50ad-49a9-ba37-70a59b58c8c4","Type":"ContainerStarted","Data":"fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87"} Nov 01 22:37:35.055314 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.055218 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" event={"ID":"f08dabd9-50ad-49a9-ba37-70a59b58c8c4","Type":"ContainerStarted","Data":"9425269bad321aff725101d2110e48241fd44dd7f46939ec4f1ea80e0edce9dd"} Nov 01 22:37:35.056055 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.056035 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:35.066572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.066529 6287 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9nqs9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:37:35.066693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.066603 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" podUID="f08dabd9-50ad-49a9-ba37-70a59b58c8c4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:37:35.077904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.077875 6287 scope.go:122] "RemoveContainer" containerID="bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e" Nov 01 22:37:35.085708 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:37:35.080065 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e\": container with ID starting with bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e not found: ID does not exist" containerID="bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e" Nov 01 22:37:35.085708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.080204 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e"} err="failed to get container status \"bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e\": rpc error: code = NotFound desc = could not find container \"bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e\": container with ID starting with bc46252783ddefada214a80f73717504d338d4403ca736674b78c3875a80c51e not found: ID does not exist" Nov 01 22:37:35.103156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.103055 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" podStartSLOduration=2.103041361 podStartE2EDuration="2.103041361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:35.102160309 +0000 UTC m=+867.750659571" watchObservedRunningTime="2026-11-01 22:37:35.103041361 +0000 UTC m=+867.751540596" Nov 01 22:37:35.142888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.142832 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk"] Nov 01 22:37:35.153472 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.153436 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-26aaf165-openshift-default-f97d464df-vhpzk"] Nov 01 22:37:35.472944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:35.472868 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044f3498-9c73-4231-9035-449af1735d27" path="/var/lib/kubelet/pods/044f3498-9c73-4231-9035-449af1735d27/volumes" Nov 01 22:37:36.067989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:36.067959 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:37:37.656575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.656541 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws"] Nov 01 22:37:37.657227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.657213 6287 cpu_manager.go:414] "containerMap: removing container" podUID="044f3498-9c73-4231-9035-449af1735d27" containerName="istio-proxy" Nov 01 22:37:37.657280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.657229 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044f3498-9c73-4231-9035-449af1735d27" containerName="istio-proxy" Nov 01 22:37:37.657403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.657392 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="044f3498-9c73-4231-9035-449af1735d27" containerName="istio-proxy" Nov 01 22:37:37.663153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.663126 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.667078 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.666836 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:37:37.667708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.667691 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:37:37.671320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.671296 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws"] Nov 01 22:37:37.699473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.699445 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.699598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.699493 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44vb\" (UniqueName: \"kubernetes.io/projected/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-kube-api-access-g44vb\") pod \"maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.699722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.699696 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.706325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.706305 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:37.706560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.706544 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:37:37.803835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.803792 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.804167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.803884 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44vb\" (UniqueName: \"kubernetes.io/projected/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-kube-api-access-g44vb\") pod \"maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.804167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.804052 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.805587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.805555 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.821225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.819601 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.821765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.821739 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44vb\" (UniqueName: \"kubernetes.io/projected/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-kube-api-access-g44vb\") pod \"maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:37.986827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:37.986741 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:38.319720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:38.319689 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws"] Nov 01 22:37:39.125901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:39.125802 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" event={"ID":"174b97b2-b1fc-40df-96ff-17eedc8cc2b7","Type":"ContainerStarted","Data":"d044fa9964a3a8efbf77a52e314d46ef1ca04bc471705c6bc1d61c3e06cd0ca8"} Nov 01 22:37:39.125901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:39.125866 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" event={"ID":"174b97b2-b1fc-40df-96ff-17eedc8cc2b7","Type":"ContainerStarted","Data":"df359a17459ce72af39617485fb1bf18c448f86470ce27c2d96e3b31647ae04e"} Nov 01 22:37:39.146958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:39.146908 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" podStartSLOduration=2.146893381 podStartE2EDuration="2.146893381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:39.14140757 +0000 UTC m=+871.789906815" watchObservedRunningTime="2026-11-01 22:37:39.146893381 +0000 UTC m=+871.795392666" Nov 01 22:37:40.136315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:40.136281 6287 generic.go:356] "Generic (PLEG): container finished" podID="174b97b2-b1fc-40df-96ff-17eedc8cc2b7" containerID="d044fa9964a3a8efbf77a52e314d46ef1ca04bc471705c6bc1d61c3e06cd0ca8" exitCode=0 Nov 01 22:37:40.136750 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:40.136362 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" event={"ID":"174b97b2-b1fc-40df-96ff-17eedc8cc2b7","Type":"ContainerDied","Data":"d044fa9964a3a8efbf77a52e314d46ef1ca04bc471705c6bc1d61c3e06cd0ca8"} Nov 01 22:37:41.090324 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.090289 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2"] Nov 01 22:37:41.099837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.098073 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-61ce6e07-2jlw2"] Nov 01 22:37:41.227856 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.227811 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc"] Nov 01 22:37:41.228342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.228140 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" podUID="842d9100-2f7b-458d-a262-c77722176e60" containerName="istio-proxy" containerID="cri-o://1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925" gracePeriod=30 Nov 01 22:37:41.526933 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.526898 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2f28e0-5c15-4c55-887f-f808d031924d" path="/var/lib/kubelet/pods/0f2f28e0-5c15-4c55-887f-f808d031924d/volumes" Nov 01 22:37:41.563940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.563903 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:41.622957 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.622914 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-xks-ca\") pod \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " Nov 01 22:37:41.623181 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.623159 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-kube-api-access-g44vb\" (UniqueName: \"kubernetes.io/projected/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-kube-api-access-g44vb\") pod \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " Nov 01 22:37:41.623434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.623231 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-openshift-ca\") pod \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\" (UID: \"174b97b2-b1fc-40df-96ff-17eedc8cc2b7\") " Nov 01 22:37:41.624879 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.624846 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-openshift-ca" pod "174b97b2-b1fc-40df-96ff-17eedc8cc2b7" (UID: "174b97b2-b1fc-40df-96ff-17eedc8cc2b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:41.626661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.626631 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-xks-ca" pod "174b97b2-b1fc-40df-96ff-17eedc8cc2b7" (UID: "174b97b2-b1fc-40df-96ff-17eedc8cc2b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:41.627051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.627023 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-kube-api-access-g44vb" pod "174b97b2-b1fc-40df-96ff-17eedc8cc2b7" (UID: "174b97b2-b1fc-40df-96ff-17eedc8cc2b7"). InnerVolumeSpecName "kube-api-access-g44vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:41.725975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.725937 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:41.725975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.725976 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:41.725975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:41.725991 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g44vb\" (UniqueName: \"kubernetes.io/projected/174b97b2-b1fc-40df-96ff-17eedc8cc2b7-kube-api-access-g44vb\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:42.165268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:42.165240 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" Nov 01 22:37:42.165443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:42.165267 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws" event={"ID":"174b97b2-b1fc-40df-96ff-17eedc8cc2b7","Type":"ContainerDied","Data":"df359a17459ce72af39617485fb1bf18c448f86470ce27c2d96e3b31647ae04e"} Nov 01 22:37:42.165443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:42.165314 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df359a17459ce72af39617485fb1bf18c448f86470ce27c2d96e3b31647ae04e" Nov 01 22:37:45.592254 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:37:45.592216 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:37:46.450345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.450320 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:37:46.535864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.535786 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-socket\") pod \"842d9100-2f7b-458d-a262-c77722176e60\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " Nov 01 22:37:46.536020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.535911 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-certs\") pod \"842d9100-2f7b-458d-a262-c77722176e60\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " Nov 01 22:37:46.536020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.535974 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-credential-socket\") pod \"842d9100-2f7b-458d-a262-c77722176e60\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " Nov 01 22:37:46.536156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536064 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/842d9100-2f7b-458d-a262-c77722176e60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842d9100-2f7b-458d-a262-c77722176e60-istio-podinfo\") pod \"842d9100-2f7b-458d-a262-c77722176e60\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " Nov 01 22:37:46.536156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536084 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-socket" pod "842d9100-2f7b-458d-a262-c77722176e60" (UID: "842d9100-2f7b-458d-a262-c77722176e60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:46.536250 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536207 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-data\") pod \"842d9100-2f7b-458d-a262-c77722176e60\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " Nov 01 22:37:46.536287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536262 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-kube-api-access-dzfn5\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-kube-api-access-dzfn5\") pod \"842d9100-2f7b-458d-a262-c77722176e60\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " Nov 01 22:37:46.536287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536259 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-certs" pod "842d9100-2f7b-458d-a262-c77722176e60" (UID: "842d9100-2f7b-458d-a262-c77722176e60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:46.536356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536286 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-istio-token\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-istio-token\") pod \"842d9100-2f7b-458d-a262-c77722176e60\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " Nov 01 22:37:46.536356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536323 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-credential-socket" pod "842d9100-2f7b-458d-a262-c77722176e60" (UID: "842d9100-2f7b-458d-a262-c77722176e60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:46.536356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536333 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-envoy\") pod \"842d9100-2f7b-458d-a262-c77722176e60\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " Nov 01 22:37:46.536489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536454 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842d9100-2f7b-458d-a262-c77722176e60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842d9100-2f7b-458d-a262-c77722176e60-istiod-ca-cert\") pod \"842d9100-2f7b-458d-a262-c77722176e60\" (UID: \"842d9100-2f7b-458d-a262-c77722176e60\") " Nov 01 22:37:46.536555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536458 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-data" pod "842d9100-2f7b-458d-a262-c77722176e60" (UID: "842d9100-2f7b-458d-a262-c77722176e60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:46.536789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.536763 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842d9100-2f7b-458d-a262-c77722176e60-istiod-ca-cert" pod "842d9100-2f7b-458d-a262-c77722176e60" (UID: "842d9100-2f7b-458d-a262-c77722176e60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:46.537269 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.537245 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.537371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.537274 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.537371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.537291 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.537371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.537308 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842d9100-2f7b-458d-a262-c77722176e60-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.537371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.537321 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.538608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.538586 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-kube-api-access-dzfn5" pod "842d9100-2f7b-458d-a262-c77722176e60" (UID: "842d9100-2f7b-458d-a262-c77722176e60"). InnerVolumeSpecName "kube-api-access-dzfn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:46.538691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.538590 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-envoy" pod "842d9100-2f7b-458d-a262-c77722176e60" (UID: "842d9100-2f7b-458d-a262-c77722176e60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:37:46.538691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.538632 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-istio-token" pod "842d9100-2f7b-458d-a262-c77722176e60" (UID: "842d9100-2f7b-458d-a262-c77722176e60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:46.538906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.538887 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/842d9100-2f7b-458d-a262-c77722176e60-istio-podinfo" pod "842d9100-2f7b-458d-a262-c77722176e60" (UID: "842d9100-2f7b-458d-a262-c77722176e60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:37:46.639035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.639005 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842d9100-2f7b-458d-a262-c77722176e60-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.639387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.639042 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzfn5\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-kube-api-access-dzfn5\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.639387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.639053 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842d9100-2f7b-458d-a262-c77722176e60-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.639387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:46.639062 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842d9100-2f7b-458d-a262-c77722176e60-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:47.226010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.225978 6287 generic.go:356] "Generic (PLEG): container finished" podID="842d9100-2f7b-458d-a262-c77722176e60" containerID="1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925" exitCode=0 Nov 01 22:37:47.226333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.226043 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" Nov 01 22:37:47.226333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.226061 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" event={"ID":"842d9100-2f7b-458d-a262-c77722176e60","Type":"ContainerDied","Data":"1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925"} Nov 01 22:37:47.226333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.226136 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc" event={"ID":"842d9100-2f7b-458d-a262-c77722176e60","Type":"ContainerDied","Data":"b1620a69e618398c38722bd405909462b956fa0ce4813065788a5c4b7108ec7b"} Nov 01 22:37:47.226333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.226165 6287 scope.go:122] "RemoveContainer" containerID="1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925" Nov 01 22:37:47.246471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.246443 6287 scope.go:122] "RemoveContainer" containerID="1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925" Nov 01 22:37:47.247080 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:37:47.247046 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925\": container with ID starting with 1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925 not found: ID does not exist" containerID="1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925" Nov 01 22:37:47.247243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.247085 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925"} err="failed to get container status \"1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925\": rpc error: code = NotFound desc = could not find container \"1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925\": container with ID starting with 1d69893cd34f424bfe217bd172aa94e3e2d025ee515985294ac4f33a093fd925 not found: ID does not exist" Nov 01 22:37:47.268163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.268108 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc"] Nov 01 22:37:47.275568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.275541 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-61ce6e07-gw-openshift-default-57dc946969-xptxc"] Nov 01 22:37:47.474226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.474187 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842d9100-2f7b-458d-a262-c77722176e60" path="/var/lib/kubelet/pods/842d9100-2f7b-458d-a262-c77722176e60/volumes" Nov 01 22:37:47.683980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.683947 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9"] Nov 01 22:37:47.684836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.684813 6287 cpu_manager.go:414] "containerMap: removing container" podUID="174b97b2-b1fc-40df-96ff-17eedc8cc2b7" containerName="revoke-keys" Nov 01 22:37:47.684944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.684837 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174b97b2-b1fc-40df-96ff-17eedc8cc2b7" containerName="revoke-keys" Nov 01 22:37:47.684944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.684864 6287 cpu_manager.go:414] "containerMap: removing container" podUID="842d9100-2f7b-458d-a262-c77722176e60" containerName="istio-proxy" Nov 01 22:37:47.684944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.684871 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842d9100-2f7b-458d-a262-c77722176e60" containerName="istio-proxy" Nov 01 22:37:47.685103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.685090 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="174b97b2-b1fc-40df-96ff-17eedc8cc2b7" containerName="revoke-keys" Nov 01 22:37:47.685231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.685128 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="842d9100-2f7b-458d-a262-c77722176e60" containerName="istio-proxy" Nov 01 22:37:47.690714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.690696 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:37:47.700355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.700336 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-0fd159-dockercfg-qktwk\"" Nov 01 22:37:47.701845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.701823 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9"] Nov 01 22:37:47.720403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.720379 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-0fd159\"" Nov 01 22:37:47.861142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.807958 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc55462a-a012-4db1-955f-0f253626f5d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9\" (UID: \"bc55462a-a012-4db1-955f-0f253626f5d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:37:47.873137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.869894 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnnd\" (UniqueName: \"kubernetes.io/projected/bc55462a-a012-4db1-955f-0f253626f5d2-kube-api-access-rlnnd\") pod \"payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9\" (UID: \"bc55462a-a012-4db1-955f-0f253626f5d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:37:47.991532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.973131 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc55462a-a012-4db1-955f-0f253626f5d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9\" (UID: \"bc55462a-a012-4db1-955f-0f253626f5d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:37:47.991532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:47.973202 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnnd\" (UniqueName: \"kubernetes.io/projected/bc55462a-a012-4db1-955f-0f253626f5d2-kube-api-access-rlnnd\") pod \"payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9\" (UID: \"bc55462a-a012-4db1-955f-0f253626f5d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:37:48.008697 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:48.006047 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc55462a-a012-4db1-955f-0f253626f5d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9\" (UID: \"bc55462a-a012-4db1-955f-0f253626f5d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:37:48.023524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:48.023486 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnnd\" (UniqueName: \"kubernetes.io/projected/bc55462a-a012-4db1-955f-0f253626f5d2-kube-api-access-rlnnd\") pod \"payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9\" (UID: \"bc55462a-a012-4db1-955f-0f253626f5d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:37:48.040973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:48.040936 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8"] Nov 01 22:37:48.047690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:48.047660 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e55f0f1-jvqf8"] Nov 01 22:37:48.310160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:48.310043 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:37:48.706457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:48.705638 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9"] Nov 01 22:37:49.260438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:49.260364 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" event={"ID":"bc55462a-a012-4db1-955f-0f253626f5d2","Type":"ContainerStarted","Data":"57ad8e379e926ddcf01524b16f124209c193e4ebf5c018e9669797e8ad599a64"} Nov 01 22:37:49.260659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:49.260465 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" event={"ID":"bc55462a-a012-4db1-955f-0f253626f5d2","Type":"ContainerStarted","Data":"002d96abb7e4c6eeaf4fb27750f5ecd85c2cd6c9bf6ba56322a410c12e031fde"} Nov 01 22:37:49.261008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:49.260972 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:37:49.283595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:49.283523 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws"] Nov 01 22:37:49.289813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:49.289764 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" podStartSLOduration=2.289750676 podStartE2EDuration="2.289750676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:49.278485116 +0000 UTC m=+881.926984351" watchObservedRunningTime="2026-11-01 22:37:49.289750676 +0000 UTC m=+881.938249924" Nov 01 22:37:49.296935 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:49.296907 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4b83ffd-r4kws"] Nov 01 22:37:49.473860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:49.473830 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174b97b2-b1fc-40df-96ff-17eedc8cc2b7" path="/var/lib/kubelet/pods/174b97b2-b1fc-40df-96ff-17eedc8cc2b7/volumes" Nov 01 22:37:49.474472 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:37:49.474449 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc07b68-c585-4093-a1a6-ce9bec6b354a" path="/var/lib/kubelet/pods/7dc07b68-c585-4093-a1a6-ce9bec6b354a/volumes" Nov 01 22:38:00.277920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:00.277893 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:38:02.029176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:02.029125 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch"] Nov 01 22:38:02.035894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:02.035866 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ac5f79-qqjch"] Nov 01 22:38:02.041088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:02.041064 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4"] Nov 01 22:38:02.045951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:02.045928 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07daabac-g4ds4"] Nov 01 22:38:03.474969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:03.474927 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea21ba9-723d-4e2e-b4e6-88eab684fa57" path="/var/lib/kubelet/pods/dea21ba9-723d-4e2e-b4e6-88eab684fa57/volumes" Nov 01 22:38:03.476337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:03.476313 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9874117-25ff-4db8-b1ca-2cdd903c1e11" path="/var/lib/kubelet/pods/e9874117-25ff-4db8-b1ca-2cdd903c1e11/volumes" Nov 01 22:38:07.265137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.265014 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7"] Nov 01 22:38:07.270775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.270749 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.285840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.285812 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7"] Nov 01 22:38:07.320392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.320363 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istiod-ca-cert\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.320561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.320404 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-data\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.320561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.320429 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-certs\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.320667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.320576 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-token\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.320747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.320718 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-envoy\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.320873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.320855 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-podinfo\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.320917 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.320891 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6pg\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-kube-api-access-zd6pg\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.320979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.320966 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-socket\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.321072 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.321059 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-credential-socket\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.353755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.353727 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fa8cda-openshift-default-dockercfg-nvxgv\"" Nov 01 22:38:07.427934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.427890 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-credential-socket\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.428267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.428241 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istiod-ca-cert\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.428416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.428401 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-data\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.428540 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.428521 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-certs\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.428696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.428680 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-token\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.428882 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.428868 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-envoy\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.429086 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.429072 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-podinfo\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.429225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.429210 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6pg\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-kube-api-access-zd6pg\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.429371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.429357 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-socket\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.430060 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.430034 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-socket\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.481751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.481718 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-certs\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.551240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.487965 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-credential-socket\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.551240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.494678 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istiod-ca-cert\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.551240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.495005 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-data\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.551240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.514194 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-token\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.551240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.514903 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6pg\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-kube-api-access-zd6pg\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.606174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.584432 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-podinfo\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.660087 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.660044 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-envoy\") pod \"e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:07.922185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.917153 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fa8cda-openshift-default-dockercfg-nvxgv\"" Nov 01 22:38:07.922185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:07.917312 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:08.242601 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:38:08.242568 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184370a9_d837_4d2a_8e83_72e7b3f6ebee.slice/crio-fdc5979da459e58456e9ad981fe87ce84a0d2f4bfac733b93b720cd5cf0395ad WatchSource:0}: Error finding container fdc5979da459e58456e9ad981fe87ce84a0d2f4bfac733b93b720cd5cf0395ad: Status 404 returned error can't find the container with id fdc5979da459e58456e9ad981fe87ce84a0d2f4bfac733b93b720cd5cf0395ad Nov 01 22:38:08.246543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:08.246517 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7"] Nov 01 22:38:08.246629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:08.246544 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:38:08.246665 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:08.246629 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:38:08.246703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:08.246667 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:38:08.688790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:08.688756 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" event={"ID":"184370a9-d837-4d2a-8e83-72e7b3f6ebee","Type":"ContainerStarted","Data":"4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561"} Nov 01 22:38:08.689230 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:08.688814 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" event={"ID":"184370a9-d837-4d2a-8e83-72e7b3f6ebee","Type":"ContainerStarted","Data":"fdc5979da459e58456e9ad981fe87ce84a0d2f4bfac733b93b720cd5cf0395ad"} Nov 01 22:38:08.776291 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:08.776227 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" podStartSLOduration=1.7762034230000001 podStartE2EDuration="1.776203423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:08.736562874 +0000 UTC m=+901.385062110" watchObservedRunningTime="2026-11-01 22:38:08.776203423 +0000 UTC m=+901.424702658" Nov 01 22:38:08.918056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:08.917971 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:09.923783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:09.923752 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:10.710938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:10.710906 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:10.711965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:10.711942 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:38:14.027795 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:14.027764 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf"] Nov 01 22:38:14.033630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:14.033604 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-036b3107-6lkjf"] Nov 01 22:38:15.473485 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:15.473454 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c" path="/var/lib/kubelet/pods/3aca02aa-8a09-4dc5-b2ad-2fc6eacaaf7c/volumes" Nov 01 22:38:15.589986 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:38:15.589950 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:38:16.027789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:16.027761 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s"] Nov 01 22:38:16.033778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:16.033751 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f67b0c-5pm4s"] Nov 01 22:38:17.037954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.037872 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb"] Nov 01 22:38:17.043971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.043947 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.049873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.049814 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-41b8f5-openshift-default-dockercfg-h4xqb\"" Nov 01 22:38:17.056459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.056430 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb"] Nov 01 22:38:17.152357 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.152324 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-credential-socket\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.152552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.152375 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-data\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.152634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.152550 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-token\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.152634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.152609 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-certs\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.152736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.152637 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istiod-ca-cert\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.152736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.152660 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fxs\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-kube-api-access-z2fxs\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.152850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.152830 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-socket\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.152887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.152859 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-podinfo\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.153064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.153040 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-envoy\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.268699 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-envoy\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.268830 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-credential-socket\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.268881 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-data\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.269005 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-token\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.269050 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-certs\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.269077 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istiod-ca-cert\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.269106 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fxs\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-kube-api-access-z2fxs\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.269173 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-socket\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.269197 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-podinfo\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.270058 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-certs\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.270337 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-credential-socket\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.270577 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-data\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.276426 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-socket\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.280142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.278998 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istiod-ca-cert\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.330565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.326016 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fxs\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-kube-api-access-z2fxs\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.343689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.343611 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-token\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.352374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.344690 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-podinfo\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.395177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.389415 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-envoy\") pod \"e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.476554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.476515 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a48b7e-1c02-4f3a-8e14-6e4ac755011b" path="/var/lib/kubelet/pods/82a48b7e-1c02-4f3a-8e14-6e4ac755011b/volumes" Nov 01 22:38:17.663911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.663814 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:17.987664 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:38:17.987630 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d2d702_5ea3_46f5_b0ab_fe35808efdc5.slice/crio-b2f94b05608a8aa2f76f26ee800ea4ee8d01cc27880ee4762400980ad2e09831 WatchSource:0}: Error finding container b2f94b05608a8aa2f76f26ee800ea4ee8d01cc27880ee4762400980ad2e09831: Status 404 returned error can't find the container with id b2f94b05608a8aa2f76f26ee800ea4ee8d01cc27880ee4762400980ad2e09831 Nov 01 22:38:17.990322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.990295 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb"] Nov 01 22:38:17.990464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.990435 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:38:17.990509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.990497 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:38:17.990541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:17.990529 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:38:18.896008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:18.895975 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" event={"ID":"91d2d702-5ea3-46f5-b0ab-fe35808efdc5","Type":"ContainerStarted","Data":"62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5"} Nov 01 22:38:18.896553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:18.896031 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" event={"ID":"91d2d702-5ea3-46f5-b0ab-fe35808efdc5","Type":"ContainerStarted","Data":"b2f94b05608a8aa2f76f26ee800ea4ee8d01cc27880ee4762400980ad2e09831"} Nov 01 22:38:18.921397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:18.921351 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" podStartSLOduration=1.921336116 podStartE2EDuration="1.921336116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:18.916412448 +0000 UTC m=+911.564911685" watchObservedRunningTime="2026-11-01 22:38:18.921336116 +0000 UTC m=+911.569835409" Nov 01 22:38:19.664901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:19.664859 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:19.671337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:19.671307 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:19.905685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:19.905655 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:19.906560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:19.906541 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:38:20.027222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:20.027181 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n"] Nov 01 22:38:20.033154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:20.033127 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x"] Nov 01 22:38:20.038379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:20.038359 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a1aea6-7g96n"] Nov 01 22:38:20.043423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:20.043402 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d8b2-j6v7x"] Nov 01 22:38:21.476163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:21.476106 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378cb8d5-70b2-4dd3-aed1-61fc626cd5cc" path="/var/lib/kubelet/pods/378cb8d5-70b2-4dd3-aed1-61fc626cd5cc/volumes" Nov 01 22:38:21.476968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:21.476943 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de4ba9c-624a-4a12-9957-2778c9983213" path="/var/lib/kubelet/pods/7de4ba9c-624a-4a12-9957-2778c9983213/volumes" Nov 01 22:38:22.457972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.457227 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95"] Nov 01 22:38:22.492137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.490914 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:22.496793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.496768 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-41b8f5\"" Nov 01 22:38:22.525247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.525215 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95"] Nov 01 22:38:22.609911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.609869 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b3c37-2479-459b-8c93-3b00a4c9407e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95\" (UID: \"b79b3c37-2479-459b-8c93-3b00a4c9407e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:22.610474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.610441 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qg82\" (UniqueName: \"kubernetes.io/projected/b79b3c37-2479-459b-8c93-3b00a4c9407e-kube-api-access-5qg82\") pod \"payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95\" (UID: \"b79b3c37-2479-459b-8c93-3b00a4c9407e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:22.722082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.721990 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b3c37-2479-459b-8c93-3b00a4c9407e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95\" (UID: \"b79b3c37-2479-459b-8c93-3b00a4c9407e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:22.723616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.723588 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qg82\" (UniqueName: \"kubernetes.io/projected/b79b3c37-2479-459b-8c93-3b00a4c9407e-kube-api-access-5qg82\") pod \"payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95\" (UID: \"b79b3c37-2479-459b-8c93-3b00a4c9407e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:22.757149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.723227 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b3c37-2479-459b-8c93-3b00a4c9407e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95\" (UID: \"b79b3c37-2479-459b-8c93-3b00a4c9407e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:22.776895 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.776861 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qg82\" (UniqueName: \"kubernetes.io/projected/b79b3c37-2479-459b-8c93-3b00a4c9407e-kube-api-access-5qg82\") pod \"payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95\" (UID: \"b79b3c37-2479-459b-8c93-3b00a4c9407e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:22.835594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:22.835553 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:23.039159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:23.039106 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg"] Nov 01 22:38:23.056897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:23.056845 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b140f2-j7klg"] Nov 01 22:38:23.296177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:23.271459 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95"] Nov 01 22:38:23.542401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:23.540232 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee1eeb2-657c-40cd-b639-d151d93ca14b" path="/var/lib/kubelet/pods/bee1eeb2-657c-40cd-b639-d151d93ca14b/volumes" Nov 01 22:38:23.967052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:23.966943 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" event={"ID":"b79b3c37-2479-459b-8c93-3b00a4c9407e","Type":"ContainerStarted","Data":"474426c83eddace490ddf12be2020aa82f036e6594eda22353f5581dd50d2eda"} Nov 01 22:38:23.967052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:23.967015 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:23.967052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:23.967036 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" event={"ID":"b79b3c37-2479-459b-8c93-3b00a4c9407e","Type":"ContainerStarted","Data":"7672ab765e70356e185693551f8b1676681c9073540a4e20d64a4fe7846a7b39"} Nov 01 22:38:23.986544 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:23.986484 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" podStartSLOduration=1.98646595 podStartE2EDuration="1.98646595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:23.984976627 +0000 UTC m=+916.633475863" watchObservedRunningTime="2026-11-01 22:38:23.98646595 +0000 UTC m=+916.634965186" Nov 01 22:38:26.045884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:26.045847 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7"] Nov 01 22:38:26.051857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:26.051830 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b59beec-jrjg7"] Nov 01 22:38:27.478097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:27.478056 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76790c6-2608-495a-868b-602a737cb506" path="/var/lib/kubelet/pods/f76790c6-2608-495a-868b-602a737cb506/volumes" Nov 01 22:38:28.033079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:28.033038 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j"] Nov 01 22:38:28.039730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:28.039696 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d8da5e5-czh9j"] Nov 01 22:38:29.033797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:29.033766 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx"] Nov 01 22:38:29.039992 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:29.039960 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe83d41-5rxbx"] Nov 01 22:38:29.474582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:29.474494 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567de28c-cc74-474f-9503-fb7b06315651" path="/var/lib/kubelet/pods/567de28c-cc74-474f-9503-fb7b06315651/volumes" Nov 01 22:38:29.475176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:29.475152 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce6ee89-77f5-4483-837d-13fa77be10df" path="/var/lib/kubelet/pods/5ce6ee89-77f5-4483-837d-13fa77be10df/volumes" Nov 01 22:38:30.026377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:30.026342 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg"] Nov 01 22:38:30.032978 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:30.032953 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6139e278-lb6wg"] Nov 01 22:38:31.474577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:31.474539 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe46b38-bb6b-4803-ac64-1764daf1a252" path="/var/lib/kubelet/pods/5fe46b38-bb6b-4803-ac64-1764daf1a252/volumes" Nov 01 22:38:33.034616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:33.034578 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh"] Nov 01 22:38:33.040448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:33.040423 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17d2e78f-cdmnh"] Nov 01 22:38:33.473317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:33.473240 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d9fa6f-0b29-4e32-9206-4e267d7bf451" path="/var/lib/kubelet/pods/e4d9fa6f-0b29-4e32-9206-4e267d7bf451/volumes" Nov 01 22:38:34.982776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:34.982739 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:38:41.028056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:41.028019 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8"] Nov 01 22:38:41.033674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:41.033643 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca2f2259-f7hq8"] Nov 01 22:38:41.483215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:41.483127 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8534fc-3426-4c34-bc85-dc1dc63e5d06" path="/var/lib/kubelet/pods/fc8534fc-3426-4c34-bc85-dc1dc63e5d06/volumes" Nov 01 22:38:42.031379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:42.031345 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt"] Nov 01 22:38:42.036908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:42.036882 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe75dc32-kspzt"] Nov 01 22:38:43.043582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.043540 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt"] Nov 01 22:38:43.050214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.050182 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a27e87-7nlkt"] Nov 01 22:38:43.473531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.473443 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a442ac-8dac-4589-be51-54cd2265aa5e" path="/var/lib/kubelet/pods/23a442ac-8dac-4589-be51-54cd2265aa5e/volumes" Nov 01 22:38:43.474024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.474007 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3db5fb-3ee8-4279-8888-14b8e6cc0df4" path="/var/lib/kubelet/pods/7b3db5fb-3ee8-4279-8888-14b8e6cc0df4/volumes" Nov 01 22:38:43.737989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.737958 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp"] Nov 01 22:38:43.743871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.743844 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:43.748313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.748278 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:38:43.748564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.748549 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:38:43.752497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.752471 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp"] Nov 01 22:38:43.812840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.812806 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:38:43.813038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.813024 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:38:43.846564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.846518 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f5ade6-hlnfp\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:43.846763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.846742 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxwvw\" (UniqueName: \"kubernetes.io/projected/2b08fbbe-cd26-4b03-90c9-3d08ff469988-kube-api-access-bxwvw\") pod \"maas-api-revoke-sub-c6f5ade6-hlnfp\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:43.846828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.846813 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f5ade6-hlnfp\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:43.948584 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.948547 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f5ade6-hlnfp\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:43.948768 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.948697 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxwvw\" (UniqueName: \"kubernetes.io/projected/2b08fbbe-cd26-4b03-90c9-3d08ff469988-kube-api-access-bxwvw\") pod \"maas-api-revoke-sub-c6f5ade6-hlnfp\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:43.948768 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.948745 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f5ade6-hlnfp\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:43.949444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.949417 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6f5ade6-hlnfp\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:43.950984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.950964 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6f5ade6-hlnfp\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:43.957208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:43.957186 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxwvw\" (UniqueName: \"kubernetes.io/projected/2b08fbbe-cd26-4b03-90c9-3d08ff469988-kube-api-access-bxwvw\") pod \"maas-api-revoke-sub-c6f5ade6-hlnfp\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:44.070025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.069944 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:44.120780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.120747 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv"] Nov 01 22:38:44.128450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.128428 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:44.133817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.133788 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-425870c1\"" Nov 01 22:38:44.136097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.136071 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-425870c1-dockercfg-vzk89\"" Nov 01 22:38:44.154826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.152590 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbswc\" (UniqueName: \"kubernetes.io/projected/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-kube-api-access-bbswc\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv\" (UID: \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:44.154826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.152850 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv\" (UID: \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:44.154826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.152955 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv"] Nov 01 22:38:44.410187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.405444 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbswc\" (UniqueName: \"kubernetes.io/projected/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-kube-api-access-bbswc\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv\" (UID: \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:44.410187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.405877 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv\" (UID: \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:44.410187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.406914 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv\" (UID: \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:44.462558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.448081 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp"] Nov 01 22:38:44.462558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.457928 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbswc\" (UniqueName: \"kubernetes.io/projected/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-kube-api-access-bbswc\") pod \"payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv\" (UID: \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:44.462558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.461444 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:44.792701 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:44.792669 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv"] Nov 01 22:38:44.874094 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:38:44.874053 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ba4d7c_33a3_4f9b_80f2_d37836509ab4.slice/crio-7d82135dc1fd867777b94574f4a73ab29f2e01a1eb98fa90a1c39e607de2646c WatchSource:0}: Error finding container 7d82135dc1fd867777b94574f4a73ab29f2e01a1eb98fa90a1c39e607de2646c: Status 404 returned error can't find the container with id 7d82135dc1fd867777b94574f4a73ab29f2e01a1eb98fa90a1c39e607de2646c Nov 01 22:38:45.088863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.088827 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz"] Nov 01 22:38:45.099308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.099277 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed7ae5-hqbzz"] Nov 01 22:38:45.457000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.456965 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" event={"ID":"2b08fbbe-cd26-4b03-90c9-3d08ff469988","Type":"ContainerStarted","Data":"db7bd1adbc057b82eef9686bb7f57e8116c0480d56a590c8e1a42c67988cd5ac"} Nov 01 22:38:45.457209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.457025 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" event={"ID":"2b08fbbe-cd26-4b03-90c9-3d08ff469988","Type":"ContainerStarted","Data":"7a15b52975e1e76a1d73085d02236d91aff9b86be26fd1188a2d569d4fdc3194"} Nov 01 22:38:45.458638 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.458609 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" event={"ID":"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4","Type":"ContainerStarted","Data":"bd9c0c7b212884cdbf9254a592467560e5f1d1f54e356fd65b0558497797b8f2"} Nov 01 22:38:45.458772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.458659 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" event={"ID":"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4","Type":"ContainerStarted","Data":"7d82135dc1fd867777b94574f4a73ab29f2e01a1eb98fa90a1c39e607de2646c"} Nov 01 22:38:45.458772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.458710 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:45.473466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.473439 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd50076-b058-41af-8482-638f0fed4601" path="/var/lib/kubelet/pods/0dd50076-b058-41af-8482-638f0fed4601/volumes" Nov 01 22:38:45.480012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.479955 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" podStartSLOduration=2.479936536 podStartE2EDuration="2.479936536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:45.474877115 +0000 UTC m=+938.123376347" watchObservedRunningTime="2026-11-01 22:38:45.479936536 +0000 UTC m=+938.128435772" Nov 01 22:38:45.499044 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:45.498993 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" podStartSLOduration=1.498975279 podStartE2EDuration="1.498975279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:45.496371397 +0000 UTC m=+938.144870646" watchObservedRunningTime="2026-11-01 22:38:45.498975279 +0000 UTC m=+938.147474516" Nov 01 22:38:45.590409 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:38:45.590369 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:38:46.497935 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:46.497902 6287 generic.go:356] "Generic (PLEG): container finished" podID="2b08fbbe-cd26-4b03-90c9-3d08ff469988" containerID="db7bd1adbc057b82eef9686bb7f57e8116c0480d56a590c8e1a42c67988cd5ac" exitCode=0 Nov 01 22:38:46.498474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:46.497998 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" event={"ID":"2b08fbbe-cd26-4b03-90c9-3d08ff469988","Type":"ContainerDied","Data":"db7bd1adbc057b82eef9686bb7f57e8116c0480d56a590c8e1a42c67988cd5ac"} Nov 01 22:38:47.687055 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.687027 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:47.800650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.800564 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b08fbbe-cd26-4b03-90c9-3d08ff469988-kube-api-access-bxwvw\" (UniqueName: \"kubernetes.io/projected/2b08fbbe-cd26-4b03-90c9-3d08ff469988-kube-api-access-bxwvw\") pod \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " Nov 01 22:38:47.800809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.800650 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-xks-ca\") pod \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " Nov 01 22:38:47.800887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.800811 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-openshift-ca\") pod \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\" (UID: \"2b08fbbe-cd26-4b03-90c9-3d08ff469988\") " Nov 01 22:38:47.801167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.801135 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-openshift-ca" pod "2b08fbbe-cd26-4b03-90c9-3d08ff469988" (UID: "2b08fbbe-cd26-4b03-90c9-3d08ff469988"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:38:47.801590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.801555 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:47.802834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.802805 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-xks-ca" pod "2b08fbbe-cd26-4b03-90c9-3d08ff469988" (UID: "2b08fbbe-cd26-4b03-90c9-3d08ff469988"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:38:47.803323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.803296 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b08fbbe-cd26-4b03-90c9-3d08ff469988-kube-api-access-bxwvw" pod "2b08fbbe-cd26-4b03-90c9-3d08ff469988" (UID: "2b08fbbe-cd26-4b03-90c9-3d08ff469988"). InnerVolumeSpecName "kube-api-access-bxwvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:47.902676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.902636 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxwvw\" (UniqueName: \"kubernetes.io/projected/2b08fbbe-cd26-4b03-90c9-3d08ff469988-kube-api-access-bxwvw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:47.902676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:47.902669 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08fbbe-cd26-4b03-90c9-3d08ff469988-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:48.520536 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:48.520507 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" Nov 01 22:38:48.520713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:48.520588 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp" event={"ID":"2b08fbbe-cd26-4b03-90c9-3d08ff469988","Type":"ContainerDied","Data":"7a15b52975e1e76a1d73085d02236d91aff9b86be26fd1188a2d569d4fdc3194"} Nov 01 22:38:48.520713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:48.520632 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a15b52975e1e76a1d73085d02236d91aff9b86be26fd1188a2d569d4fdc3194" Nov 01 22:38:51.049993 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:51.049947 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm"] Nov 01 22:38:51.058842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:51.058808 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67d9e1-57mxm"] Nov 01 22:38:51.480441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:51.478352 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef16090b-a3af-4a95-bcb8-fcff2385c39a" path="/var/lib/kubelet/pods/ef16090b-a3af-4a95-bcb8-fcff2385c39a/volumes" Nov 01 22:38:53.030394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:53.030353 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x"] Nov 01 22:38:53.040604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:53.040569 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9782305-bg47x"] Nov 01 22:38:53.531317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:53.531272 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c345eedb-8a1b-43db-8680-9aa4e2cac4f2" path="/var/lib/kubelet/pods/c345eedb-8a1b-43db-8680-9aa4e2cac4f2/volumes" Nov 01 22:38:56.093685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:56.091464 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86"] Nov 01 22:38:56.129375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:56.129333 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3a9811-9jd86"] Nov 01 22:38:56.531862 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:56.531832 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:38:57.522225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:57.516454 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da65e844-1634-4d66-abce-b559ee60d4a2" path="/var/lib/kubelet/pods/da65e844-1634-4d66-abce-b559ee60d4a2/volumes" Nov 01 22:38:59.036268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:59.036228 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4"] Nov 01 22:38:59.043979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:59.043932 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2777f2-bjgm4"] Nov 01 22:38:59.475215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:38:59.475134 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d4f8f6-8e81-4b83-87aa-fb8e280edc32" path="/var/lib/kubelet/pods/53d4f8f6-8e81-4b83-87aa-fb8e280edc32/volumes" Nov 01 22:39:04.102683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:04.102642 6287 generic.go:356] "Generic (PLEG): container finished" podID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerID="62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5" exitCode=0 Nov 01 22:39:04.103255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:04.102711 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" event={"ID":"91d2d702-5ea3-46f5-b0ab-fe35808efdc5","Type":"ContainerDied","Data":"62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5"} Nov 01 22:39:04.103319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:04.103264 6287 scope.go:122] "RemoveContainer" containerID="62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5" Nov 01 22:39:04.105188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:04.105156 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:39:04.105287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:04.105236 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:39:04.105287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:04.105267 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:39:04.905946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:04.905913 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:39:05.119745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.119706 6287 generic.go:356] "Generic (PLEG): container finished" podID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerID="4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561" exitCode=0 Nov 01 22:39:05.120222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.119774 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" event={"ID":"184370a9-d837-4d2a-8e83-72e7b3f6ebee","Type":"ContainerDied","Data":"4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561"} Nov 01 22:39:05.120363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.120330 6287 scope.go:122] "RemoveContainer" containerID="4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561" Nov 01 22:39:05.122389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.122370 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" event={"ID":"91d2d702-5ea3-46f5-b0ab-fe35808efdc5","Type":"ContainerStarted","Data":"107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720"} Nov 01 22:39:05.122494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.122382 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:39:05.122494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.122469 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:39:05.122572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.122519 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:39:05.681138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.675474 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:39:05.681138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.675540 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:39:05.681138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:05.675937 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:39:06.134170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:06.134103 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" event={"ID":"184370a9-d837-4d2a-8e83-72e7b3f6ebee","Type":"ContainerStarted","Data":"2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895"} Nov 01 22:39:06.668640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:06.668588 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:39:06.669024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:06.668990 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:39:06.929441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:06.929345 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:39:06.934270 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:06.934233 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:39:06.934411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:06.934305 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:39:07.664989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:07.664955 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:39:07.665406 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:07.665013 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:39:07.920404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:07.919494 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:39:07.920404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:07.919557 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:39:08.632223 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:08.632194 6287 scope.go:122] "RemoveContainer" containerID="19003558e4e20785aa920409ad820573978d3fcc57b3297cec8451aa2a2f6fb9" Nov 01 22:39:08.644045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:08.644022 6287 scope.go:122] "RemoveContainer" containerID="bc405dd19931f1a4c3cb847dbba0195750e4f3b33e00efec89d45911e4ff1e33" Nov 01 22:39:08.656409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:08.656388 6287 scope.go:122] "RemoveContainer" containerID="29d9c58ec62c015448351608e1b71866bf24d87b06f2e222f4ec2675547dd6db" Nov 01 22:39:08.668473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:08.668140 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:39:08.668473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:08.668197 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:39:08.921195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:08.920556 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:39:08.921195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:08.920637 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:39:09.033037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:09.032801 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c"] Nov 01 22:39:09.039166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:09.039131 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-111df7ad-l5j9c"] Nov 01 22:39:09.526133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:09.525383 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874c0e5a-ade2-4aa8-8265-6a604777fc8d" path="/var/lib/kubelet/pods/874c0e5a-ade2-4aa8-8265-6a604777fc8d/volumes" Nov 01 22:39:09.664734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:09.664701 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:39:09.664980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:09.664956 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:39:09.926207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:09.918643 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:39:09.926207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:09.918746 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:39:10.665095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:10.665060 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:39:10.665309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:10.665176 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:39:10.720437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:10.720238 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:39:10.930370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:10.920197 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:39:10.930370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:10.920265 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:39:11.674500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:11.674468 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:39:11.676510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:11.676485 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:39:11.700139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:11.686791 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:39:11.918459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:11.918424 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.918645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:11.918479 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 22:39:12.968137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:12.967454 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:39:12.980610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:12.980580 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:39:15.768975 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:39:15.768936 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:39:19.176192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.176151 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj"] Nov 01 22:39:19.178818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.178788 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2b08fbbe-cd26-4b03-90c9-3d08ff469988" containerName="revoke-keys" Nov 01 22:39:19.179037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.179019 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b08fbbe-cd26-4b03-90c9-3d08ff469988" containerName="revoke-keys" Nov 01 22:39:19.179453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.179431 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b08fbbe-cd26-4b03-90c9-3d08ff469988" containerName="revoke-keys" Nov 01 22:39:19.189444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.189413 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj"] Nov 01 22:39:19.189762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.189736 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.194244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.194213 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:39:19.194244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.194235 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:39:19.194427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.194308 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:39:19.194427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.194345 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:39:19.235179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.235136 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8tqs\" (UniqueName: \"kubernetes.io/projected/b81c0177-5394-4327-9705-73fc8b1ac2ca-kube-api-access-d8tqs\") pod \"maas-api-revoke-sub-e8bd2b4f-cx7rj\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.235336 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.235217 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8bd2b4f-cx7rj\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.235459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.235427 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8bd2b4f-cx7rj\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.337279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.337242 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8bd2b4f-cx7rj\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.337487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.337472 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tqs\" (UniqueName: \"kubernetes.io/projected/b81c0177-5394-4327-9705-73fc8b1ac2ca-kube-api-access-d8tqs\") pod \"maas-api-revoke-sub-e8bd2b4f-cx7rj\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.337555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.337538 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8bd2b4f-cx7rj\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.337997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.337966 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8bd2b4f-cx7rj\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.339951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.339930 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8bd2b4f-cx7rj\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.346997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.346974 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tqs\" (UniqueName: \"kubernetes.io/projected/b81c0177-5394-4327-9705-73fc8b1ac2ca-kube-api-access-d8tqs\") pod \"maas-api-revoke-sub-e8bd2b4f-cx7rj\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.511917 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.511886 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:19.875810 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:39:19.875763 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81c0177_5394_4327_9705_73fc8b1ac2ca.slice/crio-49258631668c61d393dc6d2cfe3b00e3ed88277a853bfc0276e0aeb821ab29ce WatchSource:0}: Error finding container 49258631668c61d393dc6d2cfe3b00e3ed88277a853bfc0276e0aeb821ab29ce: Status 404 returned error can't find the container with id 49258631668c61d393dc6d2cfe3b00e3ed88277a853bfc0276e0aeb821ab29ce Nov 01 22:39:19.878261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:19.878239 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj"] Nov 01 22:39:20.602674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:20.602640 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" event={"ID":"b81c0177-5394-4327-9705-73fc8b1ac2ca","Type":"ContainerStarted","Data":"67a4fd95e412ba893f0842d8cbb8493c6473416ed59aaba7d857b46bf3fdd143"} Nov 01 22:39:20.603071 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:20.602697 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" event={"ID":"b81c0177-5394-4327-9705-73fc8b1ac2ca","Type":"ContainerStarted","Data":"49258631668c61d393dc6d2cfe3b00e3ed88277a853bfc0276e0aeb821ab29ce"} Nov 01 22:39:20.627459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:20.627396 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" podStartSLOduration=1.627380655 podStartE2EDuration="1.627380655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:20.619266488 +0000 UTC m=+973.267765726" watchObservedRunningTime="2026-11-01 22:39:20.627380655 +0000 UTC m=+973.275879890" Nov 01 22:39:21.629226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:21.629194 6287 generic.go:356] "Generic (PLEG): container finished" podID="b81c0177-5394-4327-9705-73fc8b1ac2ca" containerID="67a4fd95e412ba893f0842d8cbb8493c6473416ed59aaba7d857b46bf3fdd143" exitCode=0 Nov 01 22:39:21.629613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:21.629279 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" event={"ID":"b81c0177-5394-4327-9705-73fc8b1ac2ca","Type":"ContainerDied","Data":"67a4fd95e412ba893f0842d8cbb8493c6473416ed59aaba7d857b46bf3fdd143"} Nov 01 22:39:22.878885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:22.878844 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:22.997315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:22.997285 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81c0177-5394-4327-9705-73fc8b1ac2ca-kube-api-access-d8tqs\" (UniqueName: \"kubernetes.io/projected/b81c0177-5394-4327-9705-73fc8b1ac2ca-kube-api-access-d8tqs\") pod \"b81c0177-5394-4327-9705-73fc8b1ac2ca\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " Nov 01 22:39:22.997561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:22.997541 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-openshift-ca\") pod \"b81c0177-5394-4327-9705-73fc8b1ac2ca\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " Nov 01 22:39:22.997690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:22.997660 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-xks-ca\") pod \"b81c0177-5394-4327-9705-73fc8b1ac2ca\" (UID: \"b81c0177-5394-4327-9705-73fc8b1ac2ca\") " Nov 01 22:39:22.997940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:22.997918 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-openshift-ca" pod "b81c0177-5394-4327-9705-73fc8b1ac2ca" (UID: "b81c0177-5394-4327-9705-73fc8b1ac2ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:22.998306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:22.998267 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:23.000499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:23.000466 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-xks-ca" pod "b81c0177-5394-4327-9705-73fc8b1ac2ca" (UID: "b81c0177-5394-4327-9705-73fc8b1ac2ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:23.000499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:23.000478 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81c0177-5394-4327-9705-73fc8b1ac2ca-kube-api-access-d8tqs" pod "b81c0177-5394-4327-9705-73fc8b1ac2ca" (UID: "b81c0177-5394-4327-9705-73fc8b1ac2ca"). InnerVolumeSpecName "kube-api-access-d8tqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:23.099992 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:23.099957 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b81c0177-5394-4327-9705-73fc8b1ac2ca-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:23.100207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:23.100007 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8tqs\" (UniqueName: \"kubernetes.io/projected/b81c0177-5394-4327-9705-73fc8b1ac2ca-kube-api-access-d8tqs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:23.652772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:23.652743 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" Nov 01 22:39:23.653010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:23.652777 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj" event={"ID":"b81c0177-5394-4327-9705-73fc8b1ac2ca","Type":"ContainerDied","Data":"49258631668c61d393dc6d2cfe3b00e3ed88277a853bfc0276e0aeb821ab29ce"} Nov 01 22:39:23.653010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:23.652828 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49258631668c61d393dc6d2cfe3b00e3ed88277a853bfc0276e0aeb821ab29ce" Nov 01 22:39:27.034064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.034028 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk"] Nov 01 22:39:27.039612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.039581 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e701836c-gjfzk"] Nov 01 22:39:27.044749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.044720 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs"] Nov 01 22:39:27.045571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.045549 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b81c0177-5394-4327-9705-73fc8b1ac2ca" containerName="revoke-keys" Nov 01 22:39:27.045666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.045571 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81c0177-5394-4327-9705-73fc8b1ac2ca" containerName="revoke-keys" Nov 01 22:39:27.045840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.045825 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81c0177-5394-4327-9705-73fc8b1ac2ca" containerName="revoke-keys" Nov 01 22:39:27.049753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.049733 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.054996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.054969 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:39:27.055167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.055033 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:39:27.056016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.055990 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:39:27.066166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.056283 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:39:27.066166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.059254 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs"] Nov 01 22:39:27.174170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.174105 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.174427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.174200 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.174427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.174249 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhct8\" (UniqueName: \"kubernetes.io/projected/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-kube-api-access-bhct8\") pod \"maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.275872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.275842 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.276080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.275891 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.276080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.275930 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhct8\" (UniqueName: \"kubernetes.io/projected/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-kube-api-access-bhct8\") pod \"maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.276579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.276553 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.278143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.278125 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.287031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.286978 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhct8\" (UniqueName: \"kubernetes.io/projected/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-kube-api-access-bhct8\") pod \"maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.372093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.372064 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:27.477425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.477390 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c" path="/var/lib/kubelet/pods/7e84b0a5-504f-49ad-be1a-efbbe8ad8a2c/volumes" Nov 01 22:39:27.656436 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:39:27.656396 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58913a9_9411_47ba_b8b0_07fd1f1a47e7.slice/crio-956574f47984c029e80e7cc6d1c130ee632e3f17afe4dc623fd7d5c3308e72ed WatchSource:0}: Error finding container 956574f47984c029e80e7cc6d1c130ee632e3f17afe4dc623fd7d5c3308e72ed: Status 404 returned error can't find the container with id 956574f47984c029e80e7cc6d1c130ee632e3f17afe4dc623fd7d5c3308e72ed Nov 01 22:39:27.658255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.658195 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs"] Nov 01 22:39:27.697629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:27.697596 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" event={"ID":"f58913a9-9411-47ba-b8b0-07fd1f1a47e7","Type":"ContainerStarted","Data":"956574f47984c029e80e7cc6d1c130ee632e3f17afe4dc623fd7d5c3308e72ed"} Nov 01 22:39:28.718202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:28.710735 6287 generic.go:356] "Generic (PLEG): container finished" podID="f58913a9-9411-47ba-b8b0-07fd1f1a47e7" containerID="c9a0258f97eb41acf4a79679a30b6af4519ae5730444915e57529d437707ffde" exitCode=0 Nov 01 22:39:28.718202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:28.710779 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" event={"ID":"f58913a9-9411-47ba-b8b0-07fd1f1a47e7","Type":"ContainerDied","Data":"c9a0258f97eb41acf4a79679a30b6af4519ae5730444915e57529d437707ffde"} Nov 01 22:39:29.903176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:29.903148 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:29.959495 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:29.959462 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-kube-api-access-bhct8\" (UniqueName: \"kubernetes.io/projected/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-kube-api-access-bhct8\") pod \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " Nov 01 22:39:29.959682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:29.959663 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-openshift-ca\") pod \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " Nov 01 22:39:29.959759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:29.959743 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-xks-ca\") pod \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\" (UID: \"f58913a9-9411-47ba-b8b0-07fd1f1a47e7\") " Nov 01 22:39:29.960026 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:29.960002 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-openshift-ca" pod "f58913a9-9411-47ba-b8b0-07fd1f1a47e7" (UID: "f58913a9-9411-47ba-b8b0-07fd1f1a47e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:29.960554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:29.960524 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:29.961677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:29.961654 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-kube-api-access-bhct8" pod "f58913a9-9411-47ba-b8b0-07fd1f1a47e7" (UID: "f58913a9-9411-47ba-b8b0-07fd1f1a47e7"). InnerVolumeSpecName "kube-api-access-bhct8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:29.961901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:29.961870 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-xks-ca" pod "f58913a9-9411-47ba-b8b0-07fd1f1a47e7" (UID: "f58913a9-9411-47ba-b8b0-07fd1f1a47e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:30.061580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:30.061546 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:30.061580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:30.061574 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhct8\" (UniqueName: \"kubernetes.io/projected/f58913a9-9411-47ba-b8b0-07fd1f1a47e7-kube-api-access-bhct8\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:30.732681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:30.732645 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" Nov 01 22:39:30.732858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:30.732678 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs" event={"ID":"f58913a9-9411-47ba-b8b0-07fd1f1a47e7","Type":"ContainerDied","Data":"956574f47984c029e80e7cc6d1c130ee632e3f17afe4dc623fd7d5c3308e72ed"} Nov 01 22:39:30.732858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:30.732729 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="956574f47984c029e80e7cc6d1c130ee632e3f17afe4dc623fd7d5c3308e72ed" Nov 01 22:39:41.026097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:41.026064 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn"] Nov 01 22:39:41.031639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:41.031609 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1729b4-f6xbn"] Nov 01 22:39:41.473434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:41.473340 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3241b4f0-0d7b-4454-99d6-29697f4b84f6" path="/var/lib/kubelet/pods/3241b4f0-0d7b-4454-99d6-29697f4b84f6/volumes" Nov 01 22:39:42.492723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:42.492674 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs"] Nov 01 22:39:42.499815 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:42.499778 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66033bd2-hjhxs"] Nov 01 22:39:43.478553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:43.478521 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58913a9-9411-47ba-b8b0-07fd1f1a47e7" path="/var/lib/kubelet/pods/f58913a9-9411-47ba-b8b0-07fd1f1a47e7/volumes" Nov 01 22:39:45.591524 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:39:45.591480 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:39:58.052368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.052284 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj"] Nov 01 22:39:58.053206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.053183 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f58913a9-9411-47ba-b8b0-07fd1f1a47e7" containerName="revoke-keys" Nov 01 22:39:58.053302 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.053208 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58913a9-9411-47ba-b8b0-07fd1f1a47e7" containerName="revoke-keys" Nov 01 22:39:58.053355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.053345 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58913a9-9411-47ba-b8b0-07fd1f1a47e7" containerName="revoke-keys" Nov 01 22:39:58.056924 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.056907 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:39:58.069166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.069100 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-93d36a\"" Nov 01 22:39:58.069275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.069191 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-93d36a-dockercfg-k5sds\"" Nov 01 22:39:58.077367 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.077340 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj"] Nov 01 22:39:58.236147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.234140 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlqf6\" (UniqueName: \"kubernetes.io/projected/558e040b-84a9-47f0-a2b9-edcde5995e68-kube-api-access-wlqf6\") pod \"payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj\" (UID: \"558e040b-84a9-47f0-a2b9-edcde5995e68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:39:58.236147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.234263 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558e040b-84a9-47f0-a2b9-edcde5995e68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj\" (UID: \"558e040b-84a9-47f0-a2b9-edcde5995e68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:39:58.351296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.351194 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlqf6\" (UniqueName: \"kubernetes.io/projected/558e040b-84a9-47f0-a2b9-edcde5995e68-kube-api-access-wlqf6\") pod \"payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj\" (UID: \"558e040b-84a9-47f0-a2b9-edcde5995e68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:39:58.351971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.351946 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558e040b-84a9-47f0-a2b9-edcde5995e68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj\" (UID: \"558e040b-84a9-47f0-a2b9-edcde5995e68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:39:58.353137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.352860 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558e040b-84a9-47f0-a2b9-edcde5995e68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj\" (UID: \"558e040b-84a9-47f0-a2b9-edcde5995e68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:39:58.405080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.405049 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlqf6\" (UniqueName: \"kubernetes.io/projected/558e040b-84a9-47f0-a2b9-edcde5995e68-kube-api-access-wlqf6\") pod \"payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj\" (UID: \"558e040b-84a9-47f0-a2b9-edcde5995e68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:39:58.672165 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:58.672055 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:39:59.002058 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:39:59.002023 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558e040b_84a9_47f0_a2b9_edcde5995e68.slice/crio-f7e55118feb881195c682faa6334fdd9e273d9135cb04ba96c08cc5d9d676c93 WatchSource:0}: Error finding container f7e55118feb881195c682faa6334fdd9e273d9135cb04ba96c08cc5d9d676c93: Status 404 returned error can't find the container with id f7e55118feb881195c682faa6334fdd9e273d9135cb04ba96c08cc5d9d676c93 Nov 01 22:39:59.003696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:59.003667 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj"] Nov 01 22:39:59.368929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:59.368777 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" event={"ID":"558e040b-84a9-47f0-a2b9-edcde5995e68","Type":"ContainerStarted","Data":"6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269"} Nov 01 22:39:59.368929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:59.368847 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" event={"ID":"558e040b-84a9-47f0-a2b9-edcde5995e68","Type":"ContainerStarted","Data":"f7e55118feb881195c682faa6334fdd9e273d9135cb04ba96c08cc5d9d676c93"} Nov 01 22:39:59.369463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:59.369033 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:39:59.390084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:39:59.390027 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" podStartSLOduration=1.390012781 podStartE2EDuration="1.390012781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:59.386504854 +0000 UTC m=+1012.035004089" watchObservedRunningTime="2026-11-01 22:39:59.390012781 +0000 UTC m=+1012.038512015" Nov 01 22:40:00.033603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:00.033571 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd"] Nov 01 22:40:00.038786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:00.038761 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56f0bf-5bjxd"] Nov 01 22:40:01.504668 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:01.501094 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db502368-1c36-415a-b880-0f87ad31f871" path="/var/lib/kubelet/pods/db502368-1c36-415a-b880-0f87ad31f871/volumes" Nov 01 22:40:04.027298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:04.027261 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9"] Nov 01 22:40:04.035202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:04.035170 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62292fe6-zlfk9"] Nov 01 22:40:05.481326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:05.481291 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8504a501-0a9d-4127-b42e-e228b2767edd" path="/var/lib/kubelet/pods/8504a501-0a9d-4127-b42e-e228b2767edd/volumes" Nov 01 22:40:08.771144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.771097 6287 scope.go:122] "RemoveContainer" containerID="39855cfb735256b2770b2c2dc663d3ba4d0b9da25cba1f1d4daaa92adde53a6c" Nov 01 22:40:08.783189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.783170 6287 scope.go:122] "RemoveContainer" containerID="8ab2e8b348ac5c76682b39f5f1d956f371d5f6da46d974a57505c826547d246a" Nov 01 22:40:08.794541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.794517 6287 scope.go:122] "RemoveContainer" containerID="71f29b856afd2a40ae6bd68391a24e1a9a1325b6ec64edf473e36a74f0159d47" Nov 01 22:40:08.805634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.805613 6287 scope.go:122] "RemoveContainer" containerID="c3d99a07e5cf4c4f5b543f84db3efac5d7d73b7972dd944294e78a174567b39e" Nov 01 22:40:08.816642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.816617 6287 scope.go:122] "RemoveContainer" containerID="8fdb1fedd06125644e8961a8c0aaee078d833deb33f35159457f4efbeddedc80" Nov 01 22:40:08.827529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.827506 6287 scope.go:122] "RemoveContainer" containerID="863b3376d2d1119414737f73feb6d0ab88abfa61d5142ba3e8b40589e8455fce" Nov 01 22:40:08.838447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.838429 6287 scope.go:122] "RemoveContainer" containerID="3ed4580f398e75b90f82817d2768372a942d170a3b44d3426f6ec6e611f4623e" Nov 01 22:40:08.850094 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.850059 6287 scope.go:122] "RemoveContainer" containerID="6a8dc3d9f94e3e57a048436c31c073b24a57340f1d1876b6664c4d0bbc9a02b1" Nov 01 22:40:08.863902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.863875 6287 scope.go:122] "RemoveContainer" containerID="372b7e2ada4246e969ac01b0631960407e2fd274f148d999cc98d802bc6b0eea" Nov 01 22:40:08.872523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.872499 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq"] Nov 01 22:40:08.876246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.875979 6287 scope.go:122] "RemoveContainer" containerID="3263f907320b021f87886779cced718ea84ba716e3debb2ff697179afe3e8d7a" Nov 01 22:40:08.878520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.878497 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:08.891657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.886834 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-c9a7ce\"" Nov 01 22:40:08.905410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.899400 6287 scope.go:122] "RemoveContainer" containerID="e0ebe3afc7fbff9e6fe0a08e25cf5a9818e673ae34dca00a0c601d990f916b53" Nov 01 22:40:08.929901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.929745 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-c9a7ce-dockercfg-v4hvp\"" Nov 01 22:40:08.940826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.940782 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq"] Nov 01 22:40:08.944879 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.944847 6287 scope.go:122] "RemoveContainer" containerID="95481a4779806e911abf76f276baefdf70dd5ee410abc151d6f800c91664d52c" Nov 01 22:40:08.947173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.947151 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng"] Nov 01 22:40:08.952217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.952193 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:08.952969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.952948 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng"] Nov 01 22:40:08.959191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.959167 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c99002-ccd6-492a-b69a-198d9cb9503f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq\" (UID: \"c6c99002-ccd6-492a-b69a-198d9cb9503f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:08.959297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.959210 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwktx\" (UniqueName: \"kubernetes.io/projected/c6c99002-ccd6-492a-b69a-198d9cb9503f-kube-api-access-zwktx\") pod \"payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq\" (UID: \"c6c99002-ccd6-492a-b69a-198d9cb9503f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:08.960048 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.960029 6287 scope.go:122] "RemoveContainer" containerID="acdbe1114d7c43f56e0402d45e878bd71a840d89cf031ae36d88bfabf3afd951" Nov 01 22:40:08.976909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.976882 6287 scope.go:122] "RemoveContainer" containerID="92d7865bb78c2387e24101e9465077c0fbeda191cb2b8936fdbc0ec15b3000f3" Nov 01 22:40:08.987624 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:08.987604 6287 scope.go:122] "RemoveContainer" containerID="ddc97e5497aa7e373793386a2614fc785fba9b6e20563e8ed16a8ddafbbb68fd" Nov 01 22:40:09.000619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.000598 6287 scope.go:122] "RemoveContainer" containerID="07c30c610a77e09c10576a2b9ea76dba54ba3978516a5362442ff16daa8ca230" Nov 01 22:40:09.062405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.061829 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng\" (UID: \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:09.063031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.062964 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c99002-ccd6-492a-b69a-198d9cb9503f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq\" (UID: \"c6c99002-ccd6-492a-b69a-198d9cb9503f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:09.063239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.063214 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c99002-ccd6-492a-b69a-198d9cb9503f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq\" (UID: \"c6c99002-ccd6-492a-b69a-198d9cb9503f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:09.063406 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.063388 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwktx\" (UniqueName: \"kubernetes.io/projected/c6c99002-ccd6-492a-b69a-198d9cb9503f-kube-api-access-zwktx\") pod \"payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq\" (UID: \"c6c99002-ccd6-492a-b69a-198d9cb9503f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:09.085158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.067348 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdnb\" (UniqueName: \"kubernetes.io/projected/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-kube-api-access-wtdnb\") pod \"payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng\" (UID: \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:09.086091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.085465 6287 scope.go:122] "RemoveContainer" containerID="44646e8073b3c49d11489952e9d2f8f3e921e24c2c59e2419e5b1de03a4b6004" Nov 01 22:40:09.099194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.092801 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwktx\" (UniqueName: \"kubernetes.io/projected/c6c99002-ccd6-492a-b69a-198d9cb9503f-kube-api-access-zwktx\") pod \"payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq\" (UID: \"c6c99002-ccd6-492a-b69a-198d9cb9503f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:09.180425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.176949 6287 scope.go:122] "RemoveContainer" containerID="c881e38c3555465ed424b44466e52bdf7a1369247f7654ea6328b3546a4bb2f3" Nov 01 22:40:09.180425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.180096 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdnb\" (UniqueName: \"kubernetes.io/projected/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-kube-api-access-wtdnb\") pod \"payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng\" (UID: \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:09.185660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.181492 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng\" (UID: \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:09.185660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.182487 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng\" (UID: \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:09.194143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.191699 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdnb\" (UniqueName: \"kubernetes.io/projected/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-kube-api-access-wtdnb\") pod \"payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng\" (UID: \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:09.254147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.251136 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:09.290136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.286992 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:09.295133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.293447 6287 scope.go:122] "RemoveContainer" containerID="a235e00caef896ea04d88cd008b15cbbf4173d5f4e73f9ab55b1d58099e7fe85" Nov 01 22:40:09.644612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.644588 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq"] Nov 01 22:40:09.670370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:40:09.670337 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ee5fcb_2ae6_434f_b22b_03403cd28ba3.slice/crio-8abd8ee2ac4392a8beb65689a34c955219aaf05f489b97e3f48572bb7e0aa1f8 WatchSource:0}: Error finding container 8abd8ee2ac4392a8beb65689a34c955219aaf05f489b97e3f48572bb7e0aa1f8: Status 404 returned error can't find the container with id 8abd8ee2ac4392a8beb65689a34c955219aaf05f489b97e3f48572bb7e0aa1f8 Nov 01 22:40:09.673970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:09.673925 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng"] Nov 01 22:40:10.383055 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:10.383026 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:40:10.552489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:10.552449 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" event={"ID":"47ee5fcb-2ae6-434f-b22b-03403cd28ba3","Type":"ContainerStarted","Data":"acf6580b461d3cb4c18891f2e716bc899d12f6f9f44805fcd9d46785c2a15855"} Nov 01 22:40:10.552679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:10.552514 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" event={"ID":"47ee5fcb-2ae6-434f-b22b-03403cd28ba3","Type":"ContainerStarted","Data":"8abd8ee2ac4392a8beb65689a34c955219aaf05f489b97e3f48572bb7e0aa1f8"} Nov 01 22:40:10.552679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:10.552551 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:10.554535 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:10.554504 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" event={"ID":"c6c99002-ccd6-492a-b69a-198d9cb9503f","Type":"ContainerStarted","Data":"0db3909df1d1f285372f7603a69312020fec798b5e9ae3946b5533607ea83d2d"} Nov 01 22:40:10.554641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:10.554563 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" event={"ID":"c6c99002-ccd6-492a-b69a-198d9cb9503f","Type":"ContainerStarted","Data":"f6669273328f43cacc287ff47c67fd806f00f7858aa42c7642f1b95cd42bc1bc"} Nov 01 22:40:10.554641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:10.554595 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:10.573968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:10.573919 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" podStartSLOduration=2.5739062329999998 podStartE2EDuration="2.573906233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:10.570988187 +0000 UTC m=+1023.219487445" watchObservedRunningTime="2026-11-01 22:40:10.573906233 +0000 UTC m=+1023.222405468" Nov 01 22:40:10.591840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:10.591795 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" podStartSLOduration=2.591782758 podStartE2EDuration="2.591782758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:10.590713318 +0000 UTC m=+1023.239212554" watchObservedRunningTime="2026-11-01 22:40:10.591782758 +0000 UTC m=+1023.240281993" Nov 01 22:40:12.025840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:12.025804 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b"] Nov 01 22:40:12.031165 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:12.031137 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59688282-rnx8b"] Nov 01 22:40:13.473267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:13.473236 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a583e01-b0c1-430e-bf60-9db46dd2d27b" path="/var/lib/kubelet/pods/0a583e01-b0c1-430e-bf60-9db46dd2d27b/volumes" Nov 01 22:40:15.590227 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:40:15.590189 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:40:16.048277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:16.048242 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng"] Nov 01 22:40:16.053478 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:16.053456 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6d30fde-95sng"] Nov 01 22:40:17.029309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:17.029275 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd"] Nov 01 22:40:17.034336 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:17.034311 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a9ab658-rnhcd"] Nov 01 22:40:17.475864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:17.475781 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15180b47-9230-4fe4-9938-92c0183686f5" path="/var/lib/kubelet/pods/15180b47-9230-4fe4-9938-92c0183686f5/volumes" Nov 01 22:40:17.476527 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:17.476507 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65b68d0-5705-45d7-900f-937a78fe6e7c" path="/var/lib/kubelet/pods/e65b68d0-5705-45d7-900f-937a78fe6e7c/volumes" Nov 01 22:40:21.566509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:21.566477 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:40:21.566918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:21.566540 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:40:22.817326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.817290 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht"] Nov 01 22:40:22.822332 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.822310 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:22.838092 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.838051 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht"] Nov 01 22:40:22.897330 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.897303 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-dcd5d4-openshift-default-dockercfg-5jlfp\"" Nov 01 22:40:22.954136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.946653 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:22.954136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.946823 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kjq\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-kube-api-access-z8kjq\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:22.954136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.946876 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:22.954136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.946922 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:22.954136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.947061 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:22.954136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.947130 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:22.954136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.947257 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:22.954136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.947376 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:22.954136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:22.947410 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.054260 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.054684 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.054777 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.054847 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kjq\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-kube-api-access-z8kjq\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.054894 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.054937 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.055067 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.055135 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.055277 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.055570 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.055777 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.056960 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.057910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.057245 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.058652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.058624 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.117186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.115291 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969"] Nov 01 22:40:23.138303 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.138233 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.138919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.138894 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.145208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.139690 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.167143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.149669 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kjq\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-kube-api-access-z8kjq\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.210186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.207442 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53d1339-nj969"] Nov 01 22:40:23.442790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.442703 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:23.475378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.475349 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4bba59-43df-4446-a6dd-f7891cc377fc" path="/var/lib/kubelet/pods/ff4bba59-43df-4446-a6dd-f7891cc377fc/volumes" Nov 01 22:40:23.761282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.758689 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht"] Nov 01 22:40:23.765008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.764959 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:40:23.765189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.765102 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:40:23.765260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:23.765207 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:40:24.509436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.509406 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb"] Nov 01 22:40:24.514101 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.514083 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.518318 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.518292 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:40:24.519083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.519048 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:40:24.522769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.522747 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb"] Nov 01 22:40:24.530519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.530492 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:40:24.530625 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.530538 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:24.590498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.590463 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b476e287-7l7sb\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.590671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.590630 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sx5v\" (UniqueName: \"kubernetes.io/projected/f6e98c1d-7b61-49ad-9836-542804accd90-kube-api-access-7sx5v\") pod \"maas-api-revoke-sub-b476e287-7l7sb\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.590753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.590682 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b476e287-7l7sb\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.694869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.694821 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b476e287-7l7sb\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.695067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.694991 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sx5v\" (UniqueName: \"kubernetes.io/projected/f6e98c1d-7b61-49ad-9836-542804accd90-kube-api-access-7sx5v\") pod \"maas-api-revoke-sub-b476e287-7l7sb\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.695067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.695053 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b476e287-7l7sb\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.695887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.695859 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b476e287-7l7sb\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.707545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.707516 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b476e287-7l7sb\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.709161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.709134 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sx5v\" (UniqueName: \"kubernetes.io/projected/f6e98c1d-7b61-49ad-9836-542804accd90-kube-api-access-7sx5v\") pod \"maas-api-revoke-sub-b476e287-7l7sb\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:24.737188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.737158 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" event={"ID":"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186","Type":"ContainerStarted","Data":"f9c509da9a833d89d77807d307cb416dee4072583a498238bb025c08028e4676"} Nov 01 22:40:24.737318 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.737236 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" event={"ID":"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186","Type":"ContainerStarted","Data":"bfbc0b9fd77f0e8828288bd17ecf1796e571203b4e750c03deb143c3a8f9e560"} Nov 01 22:40:24.762743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.762658 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podStartSLOduration=2.762643712 podStartE2EDuration="2.762643712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:24.756886583 +0000 UTC m=+1037.405385815" watchObservedRunningTime="2026-11-01 22:40:24.762643712 +0000 UTC m=+1037.411142948" Nov 01 22:40:24.833935 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:24.833901 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:25.179077 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:40:25.179044 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e98c1d_7b61_49ad_9836_542804accd90.slice/crio-d44c22aeaf86dcf45aa2eb5dc24554be29bf34f87b12f1d4c9d51917e66dd874 WatchSource:0}: Error finding container d44c22aeaf86dcf45aa2eb5dc24554be29bf34f87b12f1d4c9d51917e66dd874: Status 404 returned error can't find the container with id d44c22aeaf86dcf45aa2eb5dc24554be29bf34f87b12f1d4c9d51917e66dd874 Nov 01 22:40:25.182225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:25.182195 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb"] Nov 01 22:40:25.443760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:25.443681 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:25.449943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:25.449918 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:25.782183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:25.777367 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" event={"ID":"f6e98c1d-7b61-49ad-9836-542804accd90","Type":"ContainerStarted","Data":"2c2f2639ab380b613a70ccf05cb008ab2df31eabbae7467751df4eb5482f4aef"} Nov 01 22:40:25.782183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:25.777469 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:25.782183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:25.777501 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" event={"ID":"f6e98c1d-7b61-49ad-9836-542804accd90","Type":"ContainerStarted","Data":"d44c22aeaf86dcf45aa2eb5dc24554be29bf34f87b12f1d4c9d51917e66dd874"} Nov 01 22:40:25.782183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:25.778678 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:40:25.796413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:25.796355 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" podStartSLOduration=1.7963413620000002 podStartE2EDuration="1.796341362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:25.793828256 +0000 UTC m=+1038.442327492" watchObservedRunningTime="2026-11-01 22:40:25.796341362 +0000 UTC m=+1038.444840596" Nov 01 22:40:26.049203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:26.049072 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6"] Nov 01 22:40:26.074779 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:26.074747 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc42a633-8lcs6"] Nov 01 22:40:26.827971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:26.827758 6287 generic.go:356] "Generic (PLEG): container finished" podID="f6e98c1d-7b61-49ad-9836-542804accd90" containerID="2c2f2639ab380b613a70ccf05cb008ab2df31eabbae7467751df4eb5482f4aef" exitCode=0 Nov 01 22:40:26.827971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:26.827835 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" event={"ID":"f6e98c1d-7b61-49ad-9836-542804accd90","Type":"ContainerDied","Data":"2c2f2639ab380b613a70ccf05cb008ab2df31eabbae7467751df4eb5482f4aef"} Nov 01 22:40:27.048919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:27.048879 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96"] Nov 01 22:40:27.054188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:27.054158 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87e818e-gbr96"] Nov 01 22:40:27.476020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:27.475985 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bab43fc-43ee-4869-9cc7-3fcded2f4c55" path="/var/lib/kubelet/pods/2bab43fc-43ee-4869-9cc7-3fcded2f4c55/volumes" Nov 01 22:40:27.476555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:27.476529 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa21f90-4e1e-432e-b0da-4d1108390782" path="/var/lib/kubelet/pods/faa21f90-4e1e-432e-b0da-4d1108390782/volumes" Nov 01 22:40:28.006772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.006743 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:28.113094 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.113012 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e98c1d-7b61-49ad-9836-542804accd90-kube-api-access-7sx5v\" (UniqueName: \"kubernetes.io/projected/f6e98c1d-7b61-49ad-9836-542804accd90-kube-api-access-7sx5v\") pod \"f6e98c1d-7b61-49ad-9836-542804accd90\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " Nov 01 22:40:28.113283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.113192 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-xks-ca\") pod \"f6e98c1d-7b61-49ad-9836-542804accd90\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " Nov 01 22:40:28.113283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.113261 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-openshift-ca\") pod \"f6e98c1d-7b61-49ad-9836-542804accd90\" (UID: \"f6e98c1d-7b61-49ad-9836-542804accd90\") " Nov 01 22:40:28.113872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.113840 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-openshift-ca" pod "f6e98c1d-7b61-49ad-9836-542804accd90" (UID: "f6e98c1d-7b61-49ad-9836-542804accd90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:28.114308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.114280 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:28.115762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.115740 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-xks-ca" pod "f6e98c1d-7b61-49ad-9836-542804accd90" (UID: "f6e98c1d-7b61-49ad-9836-542804accd90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:28.116092 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.116072 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e98c1d-7b61-49ad-9836-542804accd90-kube-api-access-7sx5v" pod "f6e98c1d-7b61-49ad-9836-542804accd90" (UID: "f6e98c1d-7b61-49ad-9836-542804accd90"). InnerVolumeSpecName "kube-api-access-7sx5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:28.215644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.215605 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sx5v\" (UniqueName: \"kubernetes.io/projected/f6e98c1d-7b61-49ad-9836-542804accd90-kube-api-access-7sx5v\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:28.215644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.215637 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6e98c1d-7b61-49ad-9836-542804accd90-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:28.930282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.930238 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" event={"ID":"f6e98c1d-7b61-49ad-9836-542804accd90","Type":"ContainerDied","Data":"d44c22aeaf86dcf45aa2eb5dc24554be29bf34f87b12f1d4c9d51917e66dd874"} Nov 01 22:40:28.930487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.930336 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44c22aeaf86dcf45aa2eb5dc24554be29bf34f87b12f1d4c9d51917e66dd874" Nov 01 22:40:28.930487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:28.930417 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb" Nov 01 22:40:29.428035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.427998 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg"] Nov 01 22:40:29.428873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.428848 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f6e98c1d-7b61-49ad-9836-542804accd90" containerName="revoke-keys" Nov 01 22:40:29.428873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.428869 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e98c1d-7b61-49ad-9836-542804accd90" containerName="revoke-keys" Nov 01 22:40:29.429153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.429126 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e98c1d-7b61-49ad-9836-542804accd90" containerName="revoke-keys" Nov 01 22:40:29.434752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.434730 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.440175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.440103 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pshj\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-kube-api-access-6pshj\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.440316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.440275 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd434389-e042-41e4-a84f-325058069d49-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.440425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.440371 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.440425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.440416 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.440536 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.440436 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.440536 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.440462 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.440536 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.440517 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.440678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.440601 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.440678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.440650 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd434389-e042-41e4-a84f-325058069d49-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.461479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.461357 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg"] Nov 01 22:40:29.465504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.464418 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a1d4be-openshift-default-dockercfg-2snjb\"" Nov 01 22:40:29.587140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.584340 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.587140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.584425 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.587140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.584449 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.587140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.584485 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.587140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.584562 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.587140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.584699 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.587140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.584776 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd434389-e042-41e4-a84f-325058069d49-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.587140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.584892 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pshj\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-kube-api-access-6pshj\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.587140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.584928 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd434389-e042-41e4-a84f-325058069d49-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.600135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.593944 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.600135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.594246 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.600135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.594497 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.600135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.594742 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.600135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.595709 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd434389-e042-41e4-a84f-325058069d49-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.618850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.609729 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.651136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.650316 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pshj\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-kube-api-access-6pshj\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.661975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.659532 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.661975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.661876 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd434389-e042-41e4-a84f-325058069d49-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:29.764491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:29.764462 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:30.036457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:30.036374 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv"] Nov 01 22:40:30.041875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:30.041844 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360ed6d2-25lxv"] Nov 01 22:40:30.074253 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:40:30.074228 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd434389_e042_41e4_a84f_325058069d49.slice/crio-dd8e4c9ff131a466d8a92d436136e53c09922a153c0867e7f17f3ebc71e0b7bd WatchSource:0}: Error finding container dd8e4c9ff131a466d8a92d436136e53c09922a153c0867e7f17f3ebc71e0b7bd: Status 404 returned error can't find the container with id dd8e4c9ff131a466d8a92d436136e53c09922a153c0867e7f17f3ebc71e0b7bd Nov 01 22:40:30.076184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:30.076159 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg"] Nov 01 22:40:30.077996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:30.077961 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:40:30.078086 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:30.078031 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:40:30.078086 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:30.078064 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:40:30.963558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:30.963519 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" event={"ID":"dd434389-e042-41e4-a84f-325058069d49","Type":"ContainerStarted","Data":"d6db61681bcf311ee0144282ca573441c65c9a49195ad1757bb7cb832fff93c4"} Nov 01 22:40:30.963961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:30.963586 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" event={"ID":"dd434389-e042-41e4-a84f-325058069d49","Type":"ContainerStarted","Data":"dd8e4c9ff131a466d8a92d436136e53c09922a153c0867e7f17f3ebc71e0b7bd"} Nov 01 22:40:30.994823 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:30.994771 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podStartSLOduration=1.994752551 podStartE2EDuration="1.994752551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:30.990788965 +0000 UTC m=+1043.639288226" watchObservedRunningTime="2026-11-01 22:40:30.994752551 +0000 UTC m=+1043.643251781" Nov 01 22:40:31.473560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:31.473528 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b26bb0-edc8-43da-9237-5ba2598af0fd" path="/var/lib/kubelet/pods/f8b26bb0-edc8-43da-9237-5ba2598af0fd/volumes" Nov 01 22:40:31.765598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:31.765569 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:31.772003 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:31.771978 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:32.068883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:32.068636 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:32.070428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:32.070402 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:40:32.114220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:32.114182 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9"] Nov 01 22:40:32.129818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:32.129783 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37291d3-ghhr9"] Nov 01 22:40:33.473657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:33.473623 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b004e7b4-0d12-4916-95ff-16dc7ddc407d" path="/var/lib/kubelet/pods/b004e7b4-0d12-4916-95ff-16dc7ddc407d/volumes" Nov 01 22:40:34.051864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.051831 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx"] Nov 01 22:40:34.061954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.061922 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b7bf86-trkkx"] Nov 01 22:40:34.730737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.730705 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g"] Nov 01 22:40:34.736315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.736292 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:34.747780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.747757 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a1d4be\"" Nov 01 22:40:34.749024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.749000 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g"] Nov 01 22:40:34.750979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.750960 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a1d4be-dockercfg-h8659\"" Nov 01 22:40:34.756998 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.756971 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv"] Nov 01 22:40:34.761579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.761558 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:34.791217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.791193 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv"] Nov 01 22:40:34.998087 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.996762 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjxrg\" (UniqueName: \"kubernetes.io/projected/bb43505a-291d-4c54-b956-d2ae79e41263-kube-api-access-cjxrg\") pod \"payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv\" (UID: \"bb43505a-291d-4c54-b956-d2ae79e41263\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:34.998087 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.996858 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fp4\" (UniqueName: \"kubernetes.io/projected/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-kube-api-access-24fp4\") pod \"payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g\" (UID: \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:34.998087 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.997082 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g\" (UID: \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:34.998087 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:34.997135 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43505a-291d-4c54-b956-d2ae79e41263-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv\" (UID: \"bb43505a-291d-4c54-b956-d2ae79e41263\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:35.114377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.112563 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g\" (UID: \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:35.114377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.112613 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43505a-291d-4c54-b956-d2ae79e41263-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv\" (UID: \"bb43505a-291d-4c54-b956-d2ae79e41263\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:35.114377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.112820 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjxrg\" (UniqueName: \"kubernetes.io/projected/bb43505a-291d-4c54-b956-d2ae79e41263-kube-api-access-cjxrg\") pod \"payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv\" (UID: \"bb43505a-291d-4c54-b956-d2ae79e41263\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:35.114377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.112901 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fp4\" (UniqueName: \"kubernetes.io/projected/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-kube-api-access-24fp4\") pod \"payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g\" (UID: \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:35.114377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.113473 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g\" (UID: \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:35.114377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.114018 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43505a-291d-4c54-b956-d2ae79e41263-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv\" (UID: \"bb43505a-291d-4c54-b956-d2ae79e41263\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:35.126956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.124072 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fp4\" (UniqueName: \"kubernetes.io/projected/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-kube-api-access-24fp4\") pod \"payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g\" (UID: \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:35.126956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.125962 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjxrg\" (UniqueName: \"kubernetes.io/projected/bb43505a-291d-4c54-b956-d2ae79e41263-kube-api-access-cjxrg\") pod \"payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv\" (UID: \"bb43505a-291d-4c54-b956-d2ae79e41263\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:35.355998 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.355920 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:35.379600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.379576 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:35.490679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.490645 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc90e4f4-887b-4862-af1c-7d3af401e15c" path="/var/lib/kubelet/pods/dc90e4f4-887b-4862-af1c-7d3af401e15c/volumes" Nov 01 22:40:35.726639 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:40:35.726598 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c11c9a_12c9_4ecc_9466_bf1b8b1d32d0.slice/crio-3ad1e5202c67fed4ce8f060707a5d0e93d6929fb59b16e9a4a5c3df26ded5d19 WatchSource:0}: Error finding container 3ad1e5202c67fed4ce8f060707a5d0e93d6929fb59b16e9a4a5c3df26ded5d19: Status 404 returned error can't find the container with id 3ad1e5202c67fed4ce8f060707a5d0e93d6929fb59b16e9a4a5c3df26ded5d19 Nov 01 22:40:35.727769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.727736 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g"] Nov 01 22:40:35.790601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:35.790201 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv"] Nov 01 22:40:36.173167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:36.173103 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" event={"ID":"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0","Type":"ContainerStarted","Data":"a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e"} Nov 01 22:40:36.173375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:36.173191 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:36.173375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:36.173216 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" event={"ID":"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0","Type":"ContainerStarted","Data":"3ad1e5202c67fed4ce8f060707a5d0e93d6929fb59b16e9a4a5c3df26ded5d19"} Nov 01 22:40:36.175184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:36.175150 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" event={"ID":"bb43505a-291d-4c54-b956-d2ae79e41263","Type":"ContainerStarted","Data":"eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb"} Nov 01 22:40:36.175384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:36.175357 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:36.175495 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:36.175395 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" event={"ID":"bb43505a-291d-4c54-b956-d2ae79e41263","Type":"ContainerStarted","Data":"1be3376d6bd9ba229886cde92e08004b217b23b29812db76cbc183c3163182fe"} Nov 01 22:40:36.191535 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:36.191484 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" podStartSLOduration=2.191468822 podStartE2EDuration="2.191468822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:36.189475902 +0000 UTC m=+1048.837975129" watchObservedRunningTime="2026-11-01 22:40:36.191468822 +0000 UTC m=+1048.839968058" Nov 01 22:40:36.214514 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:36.214468 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" podStartSLOduration=2.214451869 podStartE2EDuration="2.214451869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:36.207757093 +0000 UTC m=+1048.856256329" watchObservedRunningTime="2026-11-01 22:40:36.214451869 +0000 UTC m=+1048.862951104" Nov 01 22:40:38.026202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.026164 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2"] Nov 01 22:40:38.031494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.031465 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd5e0d6-5jgw2"] Nov 01 22:40:38.048291 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.048264 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w"] Nov 01 22:40:38.053174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.053156 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.057976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.057172 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:40:38.057976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.057230 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:40:38.057976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.057703 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:38.058292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.058179 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:40:38.062453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.062426 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w"] Nov 01 22:40:38.099145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.098901 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-7b77w\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.099145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.098948 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6pg\" (UniqueName: \"kubernetes.io/projected/74b242f8-c90d-4446-9d42-531bc2ee9d01-kube-api-access-ml6pg\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-7b77w\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.099145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.098971 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-7b77w\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.203550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.202463 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-7b77w\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.203550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.202531 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6pg\" (UniqueName: \"kubernetes.io/projected/74b242f8-c90d-4446-9d42-531bc2ee9d01-kube-api-access-ml6pg\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-7b77w\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.203550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.202563 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-7b77w\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.206048 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.206017 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-7b77w\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.207125 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.207092 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-7b77w\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.218933 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.218908 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6pg\" (UniqueName: \"kubernetes.io/projected/74b242f8-c90d-4446-9d42-531bc2ee9d01-kube-api-access-ml6pg\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-7b77w\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.391186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.388393 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:38.944838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:38.944807 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w"] Nov 01 22:40:39.217470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:39.217434 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" event={"ID":"74b242f8-c90d-4446-9d42-531bc2ee9d01","Type":"ContainerStarted","Data":"add5669d57b0dc5edef38be66713973eb7b0a93957215d3611d43f63c7d6729a"} Nov 01 22:40:39.473897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:39.473826 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb" path="/var/lib/kubelet/pods/06a3b7f8-7d5e-46ca-9b1a-711ca971e0fb/volumes" Nov 01 22:40:40.228956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:40.228858 6287 generic.go:356] "Generic (PLEG): container finished" podID="74b242f8-c90d-4446-9d42-531bc2ee9d01" containerID="2d594fd5dc3152c73073b43ed997fd9e57ffcde650ff1e91799733690ff04b9c" exitCode=0 Nov 01 22:40:40.229347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:40.228948 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" event={"ID":"74b242f8-c90d-4446-9d42-531bc2ee9d01","Type":"ContainerDied","Data":"2d594fd5dc3152c73073b43ed997fd9e57ffcde650ff1e91799733690ff04b9c"} Nov 01 22:40:41.433028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.433003 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:41.471297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.471267 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b242f8-c90d-4446-9d42-531bc2ee9d01-kube-api-access-ml6pg\" (UniqueName: \"kubernetes.io/projected/74b242f8-c90d-4446-9d42-531bc2ee9d01-kube-api-access-ml6pg\") pod \"74b242f8-c90d-4446-9d42-531bc2ee9d01\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " Nov 01 22:40:41.471476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.471406 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-xks-ca\") pod \"74b242f8-c90d-4446-9d42-531bc2ee9d01\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " Nov 01 22:40:41.471560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.471521 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-openshift-ca\") pod \"74b242f8-c90d-4446-9d42-531bc2ee9d01\" (UID: \"74b242f8-c90d-4446-9d42-531bc2ee9d01\") " Nov 01 22:40:41.471931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.471889 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-openshift-ca" pod "74b242f8-c90d-4446-9d42-531bc2ee9d01" (UID: "74b242f8-c90d-4446-9d42-531bc2ee9d01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:41.472295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.472273 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:41.473793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.473767 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b242f8-c90d-4446-9d42-531bc2ee9d01-kube-api-access-ml6pg" pod "74b242f8-c90d-4446-9d42-531bc2ee9d01" (UID: "74b242f8-c90d-4446-9d42-531bc2ee9d01"). InnerVolumeSpecName "kube-api-access-ml6pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:41.474590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.474571 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-xks-ca" pod "74b242f8-c90d-4446-9d42-531bc2ee9d01" (UID: "74b242f8-c90d-4446-9d42-531bc2ee9d01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:41.573939 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.573910 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml6pg\" (UniqueName: \"kubernetes.io/projected/74b242f8-c90d-4446-9d42-531bc2ee9d01-kube-api-access-ml6pg\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:41.573939 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:41.573939 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b242f8-c90d-4446-9d42-531bc2ee9d01-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:42.250437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:42.250405 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" Nov 01 22:40:42.250632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:42.250434 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w" event={"ID":"74b242f8-c90d-4446-9d42-531bc2ee9d01","Type":"ContainerDied","Data":"add5669d57b0dc5edef38be66713973eb7b0a93957215d3611d43f63c7d6729a"} Nov 01 22:40:42.250632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:42.250488 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="add5669d57b0dc5edef38be66713973eb7b0a93957215d3611d43f63c7d6729a" Nov 01 22:40:43.944727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:43.944693 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4"] Nov 01 22:40:43.945782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:43.945758 6287 cpu_manager.go:414] "containerMap: removing container" podUID="74b242f8-c90d-4446-9d42-531bc2ee9d01" containerName="revoke-keys" Nov 01 22:40:43.945897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:43.945782 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b242f8-c90d-4446-9d42-531bc2ee9d01" containerName="revoke-keys" Nov 01 22:40:43.946042 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:43.946027 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b242f8-c90d-4446-9d42-531bc2ee9d01" containerName="revoke-keys" Nov 01 22:40:43.950333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:43.950312 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:43.957897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:43.957868 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-3fa400-dockercfg-fph9z\"" Nov 01 22:40:43.963032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:43.963008 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4"] Nov 01 22:40:43.979587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:43.979551 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-3fa400\"" Nov 01 22:40:44.070141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:44.064574 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05897fec-6ded-4175-9ead-46d45ce818f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4\" (UID: \"05897fec-6ded-4175-9ead-46d45ce818f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:44.070141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:44.064642 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/05897fec-6ded-4175-9ead-46d45ce818f8-kube-api-access-4vlvg\") pod \"payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4\" (UID: \"05897fec-6ded-4175-9ead-46d45ce818f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:44.202188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:44.169639 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05897fec-6ded-4175-9ead-46d45ce818f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4\" (UID: \"05897fec-6ded-4175-9ead-46d45ce818f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:44.202513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:44.175526 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05897fec-6ded-4175-9ead-46d45ce818f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4\" (UID: \"05897fec-6ded-4175-9ead-46d45ce818f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:44.202674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:44.202654 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/05897fec-6ded-4175-9ead-46d45ce818f8-kube-api-access-4vlvg\") pod \"payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4\" (UID: \"05897fec-6ded-4175-9ead-46d45ce818f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:44.269137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:44.259323 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/05897fec-6ded-4175-9ead-46d45ce818f8-kube-api-access-4vlvg\") pod \"payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4\" (UID: \"05897fec-6ded-4175-9ead-46d45ce818f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:44.270504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:44.270466 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:44.636445 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:40:44.636407 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05897fec_6ded_4175_9ead_46d45ce818f8.slice/crio-9e3a6e29dfeb2eb747a659799cef4439ad14f9983955acea8145c0a6a690d795 WatchSource:0}: Error finding container 9e3a6e29dfeb2eb747a659799cef4439ad14f9983955acea8145c0a6a690d795: Status 404 returned error can't find the container with id 9e3a6e29dfeb2eb747a659799cef4439ad14f9983955acea8145c0a6a690d795 Nov 01 22:40:44.638418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:44.638390 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4"] Nov 01 22:40:45.342145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:45.342083 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" event={"ID":"05897fec-6ded-4175-9ead-46d45ce818f8","Type":"ContainerStarted","Data":"d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025"} Nov 01 22:40:45.342552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:45.342165 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:45.342552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:45.342193 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" event={"ID":"05897fec-6ded-4175-9ead-46d45ce818f8","Type":"ContainerStarted","Data":"9e3a6e29dfeb2eb747a659799cef4439ad14f9983955acea8145c0a6a690d795"} Nov 01 22:40:45.377858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:45.377801 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" podStartSLOduration=2.377783307 podStartE2EDuration="2.377783307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:45.371896004 +0000 UTC m=+1058.020395238" watchObservedRunningTime="2026-11-01 22:40:45.377783307 +0000 UTC m=+1058.026282541" Nov 01 22:40:45.638791 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:40:45.638687 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:40:47.032815 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:47.032781 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t"] Nov 01 22:40:47.038542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:47.038515 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg"] Nov 01 22:40:47.043021 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:47.043002 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be4916a-lcf6t"] Nov 01 22:40:47.047257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:47.047228 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014bf358-dx9cg"] Nov 01 22:40:47.195202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:47.195172 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:40:47.213452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:47.213417 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:40:47.475980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:47.475902 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc" path="/var/lib/kubelet/pods/0a4f0e3b-02b8-47ed-85ec-d93b07ba9bcc/volumes" Nov 01 22:40:47.476458 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:47.476444 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9" path="/var/lib/kubelet/pods/f2415d96-c6cc-4d3d-a89d-bcdb4b356ee9/volumes" Nov 01 22:40:48.174797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:48.174759 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv"] Nov 01 22:40:52.027465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:52.027426 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z"] Nov 01 22:40:52.032312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:52.032284 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-158c7ee6-d8n7z"] Nov 01 22:40:53.176971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:53.176929 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" podUID="e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" containerName="payload-pre-processing" containerID="cri-o://bd9c0c7b212884cdbf9254a592467560e5f1d1f54e356fd65b0558497797b8f2" gracePeriod=25 Nov 01 22:40:53.474601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:53.474511 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85ea45c-e2dd-4119-a0f1-1b78c40bb867" path="/var/lib/kubelet/pods/a85ea45c-e2dd-4119-a0f1-1b78c40bb867/volumes" Nov 01 22:40:56.355313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:56.355284 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:40:56.499872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:56.499836 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" podUID="e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:40:57.094226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:57.094179 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx"] Nov 01 22:40:57.153806 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:57.153768 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ef0be5-z4cpx"] Nov 01 22:40:57.161847 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:57.161808 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk"] Nov 01 22:40:57.168863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:57.168832 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16bec70b-7x7mk"] Nov 01 22:40:57.477147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:57.477055 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555649a3-ccf4-48c6-95e7-ee7623f6ba84" path="/var/lib/kubelet/pods/555649a3-ccf4-48c6-95e7-ee7623f6ba84/volumes" Nov 01 22:40:57.477662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:57.477645 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5be362-d503-4e52-9896-844827e5f7f0" path="/var/lib/kubelet/pods/dc5be362-d503-4e52-9896-844827e5f7f0/volumes" Nov 01 22:40:58.252262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.252225 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s"] Nov 01 22:40:58.258768 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.258745 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.266486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.266463 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:40:58.266732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.266709 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:58.266856 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.266818 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:40:58.266956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.266883 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:40:58.279790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.279740 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s"] Nov 01 22:40:58.398305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.398259 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b71acd-hsk5s\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.398975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.398945 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b71acd-hsk5s\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.399952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.399930 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5lth\" (UniqueName: \"kubernetes.io/projected/fe5ca5c5-1fec-4d1b-9177-d643ad080680-kube-api-access-m5lth\") pod \"maas-api-revoke-sub-f2b71acd-hsk5s\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.535251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.532604 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b71acd-hsk5s\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.535251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.532688 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5lth\" (UniqueName: \"kubernetes.io/projected/fe5ca5c5-1fec-4d1b-9177-d643ad080680-kube-api-access-m5lth\") pod \"maas-api-revoke-sub-f2b71acd-hsk5s\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.535251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.532803 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b71acd-hsk5s\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.535251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.533743 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b71acd-hsk5s\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.556761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.554997 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b71acd-hsk5s\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.562509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.562481 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5lth\" (UniqueName: \"kubernetes.io/projected/fe5ca5c5-1fec-4d1b-9177-d643ad080680-kube-api-access-m5lth\") pod \"maas-api-revoke-sub-f2b71acd-hsk5s\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.598779 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.598751 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:40:58.895646 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:40:58.895614 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5ca5c5_1fec_4d1b_9177_d643ad080680.slice/crio-367d628883f429847942247ead20de4c8ed58a5a63afd1c80573eb835df27fa9 WatchSource:0}: Error finding container 367d628883f429847942247ead20de4c8ed58a5a63afd1c80573eb835df27fa9: Status 404 returned error can't find the container with id 367d628883f429847942247ead20de4c8ed58a5a63afd1c80573eb835df27fa9 Nov 01 22:40:58.897605 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:58.897578 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s"] Nov 01 22:40:59.564774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:59.560886 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" event={"ID":"fe5ca5c5-1fec-4d1b-9177-d643ad080680","Type":"ContainerStarted","Data":"f861ebe0bacd78cf80771f9546e5d448c89090042d5d7848870891558aa666ac"} Nov 01 22:40:59.564774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:59.560956 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" event={"ID":"fe5ca5c5-1fec-4d1b-9177-d643ad080680","Type":"ContainerStarted","Data":"367d628883f429847942247ead20de4c8ed58a5a63afd1c80573eb835df27fa9"} Nov 01 22:40:59.608798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:40:59.608738 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" podStartSLOduration=1.608720725 podStartE2EDuration="1.608720725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:59.603321156 +0000 UTC m=+1072.251820402" watchObservedRunningTime="2026-11-01 22:40:59.608720725 +0000 UTC m=+1072.257219959" Nov 01 22:41:00.576999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:00.576959 6287 generic.go:356] "Generic (PLEG): container finished" podID="fe5ca5c5-1fec-4d1b-9177-d643ad080680" containerID="f861ebe0bacd78cf80771f9546e5d448c89090042d5d7848870891558aa666ac" exitCode=0 Nov 01 22:41:00.577728 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:00.577700 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" event={"ID":"fe5ca5c5-1fec-4d1b-9177-d643ad080680","Type":"ContainerDied","Data":"f861ebe0bacd78cf80771f9546e5d448c89090042d5d7848870891558aa666ac"} Nov 01 22:41:01.799209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:01.799184 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:41:01.898892 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:01.898795 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-openshift-ca\") pod \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " Nov 01 22:41:01.899075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:01.898924 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-xks-ca\") pod \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " Nov 01 22:41:01.899160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:01.899084 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5ca5c5-1fec-4d1b-9177-d643ad080680-kube-api-access-m5lth\" (UniqueName: \"kubernetes.io/projected/fe5ca5c5-1fec-4d1b-9177-d643ad080680-kube-api-access-m5lth\") pod \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\" (UID: \"fe5ca5c5-1fec-4d1b-9177-d643ad080680\") " Nov 01 22:41:01.899232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:01.899185 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-openshift-ca" pod "fe5ca5c5-1fec-4d1b-9177-d643ad080680" (UID: "fe5ca5c5-1fec-4d1b-9177-d643ad080680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:01.899754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:01.899731 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:01.901097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:01.901074 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-xks-ca" pod "fe5ca5c5-1fec-4d1b-9177-d643ad080680" (UID: "fe5ca5c5-1fec-4d1b-9177-d643ad080680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:01.901205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:01.901090 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ca5c5-1fec-4d1b-9177-d643ad080680-kube-api-access-m5lth" pod "fe5ca5c5-1fec-4d1b-9177-d643ad080680" (UID: "fe5ca5c5-1fec-4d1b-9177-d643ad080680"). InnerVolumeSpecName "kube-api-access-m5lth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:02.001148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:02.001096 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5ca5c5-1fec-4d1b-9177-d643ad080680-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:02.001148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:02.001142 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5lth\" (UniqueName: \"kubernetes.io/projected/fe5ca5c5-1fec-4d1b-9177-d643ad080680-kube-api-access-m5lth\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:02.600166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:02.600129 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" Nov 01 22:41:02.600376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:02.600160 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s" event={"ID":"fe5ca5c5-1fec-4d1b-9177-d643ad080680","Type":"ContainerDied","Data":"367d628883f429847942247ead20de4c8ed58a5a63afd1c80573eb835df27fa9"} Nov 01 22:41:02.600376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:02.600227 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="367d628883f429847942247ead20de4c8ed58a5a63afd1c80573eb835df27fa9" Nov 01 22:41:04.188598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.188563 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2"] Nov 01 22:41:04.189404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.189387 6287 cpu_manager.go:414] "containerMap: removing container" podUID="fe5ca5c5-1fec-4d1b-9177-d643ad080680" containerName="revoke-keys" Nov 01 22:41:04.189404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.189404 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5ca5c5-1fec-4d1b-9177-d643ad080680" containerName="revoke-keys" Nov 01 22:41:04.189526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.189517 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5ca5c5-1fec-4d1b-9177-d643ad080680" containerName="revoke-keys" Nov 01 22:41:04.193373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.193355 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.234816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.234797 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ee8d1a-openshift-default-dockercfg-hsk87\"" Nov 01 22:41:04.256504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.256438 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.256666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.256638 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.256742 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.256712 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.256799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.256751 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.256901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.256883 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419eba00-1830-4c81-8c5d-912f18479177-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.257126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.257092 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419eba00-1830-4c81-8c5d-912f18479177-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.257550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.257158 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.257550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.257245 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tvdq\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-kube-api-access-5tvdq\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.257550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.257325 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.257550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.257547 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2"] Nov 01 22:41:04.396487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.396444 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.396810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.396781 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.396968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.396951 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.397262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.397237 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419eba00-1830-4c81-8c5d-912f18479177-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.397418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.397401 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419eba00-1830-4c81-8c5d-912f18479177-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.397572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.397556 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.397692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.397677 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tvdq\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-kube-api-access-5tvdq\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.397826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.397810 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.398097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.398075 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.398809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.398787 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.409196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.407517 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.409196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.407783 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.409196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.408001 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.409196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.408621 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419eba00-1830-4c81-8c5d-912f18479177-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.457186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.438195 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.485498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.483456 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419eba00-1830-4c81-8c5d-912f18479177-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.500206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.489887 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tvdq\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-kube-api-access-5tvdq\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.500206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.498007 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.511619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.510372 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:04.892622 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.892587 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2"] Nov 01 22:41:04.893219 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.893171 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:04.893325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.893304 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:04.893433 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:04.893415 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:05.766473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:05.766333 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" event={"ID":"419eba00-1830-4c81-8c5d-912f18479177","Type":"ContainerStarted","Data":"92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af"} Nov 01 22:41:05.766473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:05.766404 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" event={"ID":"419eba00-1830-4c81-8c5d-912f18479177","Type":"ContainerStarted","Data":"bed025f915506fff2f5cbcb343acba970644c4eec5b33d9f1e3acfedc5d7da7f"} Nov 01 22:41:05.817134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:05.809656 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podStartSLOduration=1.809636152 podStartE2EDuration="1.809636152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:05.808468742 +0000 UTC m=+1078.456967991" watchObservedRunningTime="2026-11-01 22:41:05.809636152 +0000 UTC m=+1078.458135388" Nov 01 22:41:06.059809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:06.059709 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph"] Nov 01 22:41:06.079142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:06.074278 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-958aa513-62rph"] Nov 01 22:41:06.511421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:06.511259 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:06.524211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:06.516848 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" podUID="e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:41:06.590670 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:06.590352 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:06.881218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:06.881043 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:06.913497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:06.913463 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:41:07.518135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:07.515766 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e4b2bc-b479-44cd-9532-f3ef6ae75452" path="/var/lib/kubelet/pods/45e4b2bc-b479-44cd-9532-f3ef6ae75452/volumes" Nov 01 22:41:09.011137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.010133 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.122208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.121433 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.254140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.250166 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.254140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.252728 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.262200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.261232 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.326444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.326387 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.349595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.349552 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.350226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.349824 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.376258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.376198 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.490381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.489216 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:41:09.526333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.526218 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-87hds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:09.526333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.526315 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.608174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.604342 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.614397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.613383 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:41:09.633696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.633664 6287 scope.go:122] "RemoveContainer" containerID="8d74f12abafa022ea1bf7baf0ebabc0a6dbceff960a5f60ddbe5dcfbd3b8ff8b" Nov 01 22:41:09.719874 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.719827 6287 scope.go:122] "RemoveContainer" containerID="5c48a6453862c73cfa201ffeaec1b6804260c7435e78dc1beb250b116e588980" Nov 01 22:41:09.735678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.734694 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:41:09.801889 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.801858 6287 scope.go:122] "RemoveContainer" containerID="0d120cd1f61e11aebe3ea422ab13e31f14041e50e9b7a91e90285f5b7fe9d041" Nov 01 22:41:09.848809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.848769 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:09.881976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:09.881230 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.032405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.032355 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.145502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.050685 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:41:10.165159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.158412 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.165159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.159186 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:41:10.265139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.260660 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.265139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.261080 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:41:10.315157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.313978 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.315157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.314329 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:41:10.315157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.314846 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:41:10.315157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.314909 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:41:10.410459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.410362 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:41:10.441660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.441626 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:41:10.491868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.491826 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.493305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.493281 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:41:10.530138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.528616 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:41:10.530138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.529524 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:41:10.566136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.538963 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.591413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.591381 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:41:10.665188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.607259 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:41:10.665188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.608889 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:41:10.665188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.628852 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.665188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.630513 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:41:10.739464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.739415 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:41:10.753783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.753731 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:41:10.870277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.870238 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.907469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.907403 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:41:10.982457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.976242 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:10.982457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.977145 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:41:10.984789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:10.984769 6287 scope.go:122] "RemoveContainer" containerID="ee84588b4e57ec6eb4f9a1a625278ffafea70aff07cff30e273d3206086ee4c6" Nov 01 22:41:11.050927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.050869 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:41:11.083900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.083870 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l"] Nov 01 22:41:11.106893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.106845 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Nov 01 22:41:11.169750 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.163330 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:41:11.269218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.266993 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.336178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.326277 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:41:11.366179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.357450 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:41:11.421139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.376355 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.472362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.412611 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:11.507613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.502189 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.609198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.515822 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 01 22:41:11.657693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.541299 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:41:11.661946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.550609 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.662842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.550937 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ee8d1a-dockercfg-7qtxc\"" Nov 01 22:41:11.694259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.551920 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ee8d1a\"" Nov 01 22:41:11.694259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.591227 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.694259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.591281 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.694259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.615188 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.694259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.622888 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l\" (UID: \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:11.694259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.676282 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjgc\" (UniqueName: \"kubernetes.io/projected/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-kube-api-access-qtjgc\") pod \"payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l\" (UID: \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:11.694259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.629181 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.866277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.851749 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:11.967326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:11.967282 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:41:12.009873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.009831 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:12.064327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.064298 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:41:12.076822 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.076797 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:41:12.077012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.076999 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:41:12.077145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.077132 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l"] Nov 01 22:41:12.077601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.077580 6287 scope.go:122] "RemoveContainer" containerID="509b2a227f00439912ece7466a836e134fc61e1e9585e4d9e05e29ee40ba1208" Nov 01 22:41:12.092191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.092154 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:12.184072 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.179596 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:41:12.258658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.258628 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:41:12.306135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.294376 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:12.306135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.300427 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:41:12.357140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.347590 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:41:12.462179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.430552 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l\" (UID: \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:12.462179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.430695 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjgc\" (UniqueName: \"kubernetes.io/projected/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-kube-api-access-qtjgc\") pod \"payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l\" (UID: \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:12.462179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.432320 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l\" (UID: \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:12.487136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.466323 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:41:12.506542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.506502 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:12.509926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.509894 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjgc\" (UniqueName: \"kubernetes.io/projected/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-kube-api-access-qtjgc\") pod \"payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l\" (UID: \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:12.581306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.581213 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:41:12.626129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.604072 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:41:12.653453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.652419 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:41:12.672076 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.670626 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:12.749138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.749085 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:12.755701 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.755669 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:41:12.863134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.842942 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:41:12.863134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.852869 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:41:12.964141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:12.961466 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:41:13.553754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:13.553673 6287 generic.go:356] "Generic (PLEG): container finished" podID="e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" containerID="bd9c0c7b212884cdbf9254a592467560e5f1d1f54e356fd65b0558497797b8f2" exitCode=0 Nov 01 22:41:13.556099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:13.555882 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" event={"ID":"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4","Type":"ContainerDied","Data":"bd9c0c7b212884cdbf9254a592467560e5f1d1f54e356fd65b0558497797b8f2"} Nov 01 22:41:13.952381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:13.950563 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l"] Nov 01 22:41:14.053315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.053273 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft"] Nov 01 22:41:14.140978 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.140946 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3595615b-9zjft"] Nov 01 22:41:14.162671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.162648 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:41:14.256365 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.256254 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-kube-api-access-bbswc\" (UniqueName: \"kubernetes.io/projected/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-kube-api-access-bbswc\") pod \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\" (UID: \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\") " Nov 01 22:41:14.256595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.256569 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-plugins-config-volume\") pod \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\" (UID: \"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4\") " Nov 01 22:41:14.257152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.257038 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-plugins-config-volume" pod "e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" (UID: "e9ba4d7c-33a3-4f9b-80f2-d37836509ab4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:14.257527 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.257494 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:14.259125 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.259082 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-kube-api-access-bbswc" pod "e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" (UID: "e9ba4d7c-33a3-4f9b-80f2-d37836509ab4"). InnerVolumeSpecName "kube-api-access-bbswc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:14.359240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.359200 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbswc\" (UniqueName: \"kubernetes.io/projected/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4-kube-api-access-bbswc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:14.569457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.569429 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" Nov 01 22:41:14.593135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.569444 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv" event={"ID":"e9ba4d7c-33a3-4f9b-80f2-d37836509ab4","Type":"ContainerDied","Data":"7d82135dc1fd867777b94574f4a73ab29f2e01a1eb98fa90a1c39e607de2646c"} Nov 01 22:41:14.593135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.590255 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" event={"ID":"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2","Type":"ContainerStarted","Data":"0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508"} Nov 01 22:41:14.593135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.590313 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" event={"ID":"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2","Type":"ContainerStarted","Data":"52ab6f2f618bcbcce8bd2fa1bb6a5f3bdb8bb5010f5c47e203f85fbc9915d999"} Nov 01 22:41:14.593135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.590346 6287 scope.go:122] "RemoveContainer" containerID="bd9c0c7b212884cdbf9254a592467560e5f1d1f54e356fd65b0558497797b8f2" Nov 01 22:41:14.593135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.590740 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:14.654986 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.654924 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" podStartSLOduration=4.654899575 podStartE2EDuration="4.654899575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:14.629931742 +0000 UTC m=+1087.278430991" watchObservedRunningTime="2026-11-01 22:41:14.654899575 +0000 UTC m=+1087.303398811" Nov 01 22:41:14.743136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.729787 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv"] Nov 01 22:41:14.760444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:14.760412 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-425870c1-595ddc7d59-7xcfv"] Nov 01 22:41:15.092139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:15.074183 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt"] Nov 01 22:41:15.105906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:15.105813 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1333ff-trjkt"] Nov 01 22:41:15.478494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:15.478402 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531d33c4-d487-41cb-bf66-99566c523e27" path="/var/lib/kubelet/pods/531d33c4-d487-41cb-bf66-99566c523e27/volumes" Nov 01 22:41:15.479154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:15.479097 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ca8b98-2067-4131-bc57-4305ff2c5485" path="/var/lib/kubelet/pods/d4ca8b98-2067-4131-bc57-4305ff2c5485/volumes" Nov 01 22:41:15.480660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:15.479620 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" path="/var/lib/kubelet/pods/e9ba4d7c-33a3-4f9b-80f2-d37836509ab4/volumes" Nov 01 22:41:15.719469 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:41:15.719430 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:41:18.117944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.116875 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw"] Nov 01 22:41:18.117944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.117723 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" containerName="payload-pre-processing" Nov 01 22:41:18.117944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.117740 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" containerName="payload-pre-processing" Nov 01 22:41:18.117944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.117889 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ba4d7c-33a3-4f9b-80f2-d37836509ab4" containerName="payload-pre-processing" Nov 01 22:41:18.122142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.120197 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:18.122142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.120257 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.244309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.244267 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw"] Nov 01 22:41:18.244513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.244326 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w"] Nov 01 22:41:18.244513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.244348 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-7b77w"] Nov 01 22:41:18.244513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.244387 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.247609 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.247582 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:41:18.247960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.247938 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:41:18.248102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.247961 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:41:18.248660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.248643 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:41:18.251104 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.250833 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bdb1e2-5g8nw\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.252581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.252552 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bdb1e2-5g8nw\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.252683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.252652 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hlx\" (UniqueName: \"kubernetes.io/projected/a6fa8f34-4795-4188-835a-08d8bf403220-kube-api-access-t6hlx\") pod \"maas-api-revoke-sub-e9bdb1e2-5g8nw\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.356534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.356490 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hlx\" (UniqueName: \"kubernetes.io/projected/a6fa8f34-4795-4188-835a-08d8bf403220-kube-api-access-t6hlx\") pod \"maas-api-revoke-sub-e9bdb1e2-5g8nw\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.356712 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.356616 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bdb1e2-5g8nw\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.356756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.356745 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bdb1e2-5g8nw\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.836730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.836624 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bdb1e2-5g8nw\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.836730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.836627 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hlx\" (UniqueName: \"kubernetes.io/projected/a6fa8f34-4795-4188-835a-08d8bf403220-kube-api-access-t6hlx\") pod \"maas-api-revoke-sub-e9bdb1e2-5g8nw\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.836730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.836625 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bdb1e2-5g8nw\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:18.926004 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:18.925967 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:19.039898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:19.039862 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch"] Nov 01 22:41:19.046085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:19.046046 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc91d391-t77ch"] Nov 01 22:41:19.530307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:19.530260 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f64e79-8ffd-45b1-82ae-54d0b562facd" path="/var/lib/kubelet/pods/54f64e79-8ffd-45b1-82ae-54d0b562facd/volumes" Nov 01 22:41:19.542145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:19.541713 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b242f8-c90d-4446-9d42-531bc2ee9d01" path="/var/lib/kubelet/pods/74b242f8-c90d-4446-9d42-531bc2ee9d01/volumes" Nov 01 22:41:20.994249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:20.994216 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw"] Nov 01 22:41:20.994915 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:41:20.994879 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fa8f34_4795_4188_835a_08d8bf403220.slice/crio-ae5a85985505e1aec3450e9edb0ab4a154bcc769be420d1ce349c3bd07253bcd WatchSource:0}: Error finding container ae5a85985505e1aec3450e9edb0ab4a154bcc769be420d1ce349c3bd07253bcd: Status 404 returned error can't find the container with id ae5a85985505e1aec3450e9edb0ab4a154bcc769be420d1ce349c3bd07253bcd Nov 01 22:41:21.803749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:21.803704 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" event={"ID":"a6fa8f34-4795-4188-835a-08d8bf403220","Type":"ContainerStarted","Data":"b8ea87abc55f42f571514fbbe780988b54d49c49ce80a80135852edf0d2e458c"} Nov 01 22:41:21.803949 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:21.803775 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" event={"ID":"a6fa8f34-4795-4188-835a-08d8bf403220","Type":"ContainerStarted","Data":"ae5a85985505e1aec3450e9edb0ab4a154bcc769be420d1ce349c3bd07253bcd"} Nov 01 22:41:21.826805 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:21.826743 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" podStartSLOduration=3.826714075 podStartE2EDuration="3.826714075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:21.823176789 +0000 UTC m=+1094.471676026" watchObservedRunningTime="2026-11-01 22:41:21.826714075 +0000 UTC m=+1094.475213309" Nov 01 22:41:22.814812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:22.814774 6287 generic.go:356] "Generic (PLEG): container finished" podID="a6fa8f34-4795-4188-835a-08d8bf403220" containerID="b8ea87abc55f42f571514fbbe780988b54d49c49ce80a80135852edf0d2e458c" exitCode=0 Nov 01 22:41:22.815298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:22.814844 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" event={"ID":"a6fa8f34-4795-4188-835a-08d8bf403220","Type":"ContainerDied","Data":"b8ea87abc55f42f571514fbbe780988b54d49c49ce80a80135852edf0d2e458c"} Nov 01 22:41:23.028974 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:23.028943 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs"] Nov 01 22:41:23.034133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:23.034092 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a093c28-4fdxs"] Nov 01 22:41:23.473333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:23.473298 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922d9d62-2412-4c75-aa1d-a6931bea1e37" path="/var/lib/kubelet/pods/922d9d62-2412-4c75-aa1d-a6931bea1e37/volumes" Nov 01 22:41:24.054159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.054101 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:24.213975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.213892 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-xks-ca\") pod \"a6fa8f34-4795-4188-835a-08d8bf403220\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " Nov 01 22:41:24.213975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.213930 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-openshift-ca\") pod \"a6fa8f34-4795-4188-835a-08d8bf403220\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " Nov 01 22:41:24.214391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.214027 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fa8f34-4795-4188-835a-08d8bf403220-kube-api-access-t6hlx\" (UniqueName: \"kubernetes.io/projected/a6fa8f34-4795-4188-835a-08d8bf403220-kube-api-access-t6hlx\") pod \"a6fa8f34-4795-4188-835a-08d8bf403220\" (UID: \"a6fa8f34-4795-4188-835a-08d8bf403220\") " Nov 01 22:41:24.214506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.214457 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-openshift-ca" pod "a6fa8f34-4795-4188-835a-08d8bf403220" (UID: "a6fa8f34-4795-4188-835a-08d8bf403220"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:24.215152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.215082 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.252874 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.252834 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-xks-ca" pod "a6fa8f34-4795-4188-835a-08d8bf403220" (UID: "a6fa8f34-4795-4188-835a-08d8bf403220"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:24.253066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.252988 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fa8f34-4795-4188-835a-08d8bf403220-kube-api-access-t6hlx" pod "a6fa8f34-4795-4188-835a-08d8bf403220" (UID: "a6fa8f34-4795-4188-835a-08d8bf403220"). InnerVolumeSpecName "kube-api-access-t6hlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:24.316891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.316860 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6hlx\" (UniqueName: \"kubernetes.io/projected/a6fa8f34-4795-4188-835a-08d8bf403220-kube-api-access-t6hlx\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.316891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.316893 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6fa8f34-4795-4188-835a-08d8bf403220-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.838093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.838045 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" event={"ID":"a6fa8f34-4795-4188-835a-08d8bf403220","Type":"ContainerDied","Data":"ae5a85985505e1aec3450e9edb0ab4a154bcc769be420d1ce349c3bd07253bcd"} Nov 01 22:41:24.838308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.838095 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw" Nov 01 22:41:24.838308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:24.838162 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae5a85985505e1aec3450e9edb0ab4a154bcc769be420d1ce349c3bd07253bcd" Nov 01 22:41:25.148497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:25.148400 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz"] Nov 01 22:41:25.153697 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:25.153672 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0beb384-77xsz"] Nov 01 22:41:25.482175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:25.481610 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176b9369-fceb-4e6b-9807-9d5cf2a21754" path="/var/lib/kubelet/pods/176b9369-fceb-4e6b-9807-9d5cf2a21754/volumes" Nov 01 22:41:25.613298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:25.613178 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:41:26.909761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:26.909711 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:26.910305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:26.909793 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:29.822783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:29.822744 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9"] Nov 01 22:41:29.892017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:29.891980 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a6fa8f34-4795-4188-835a-08d8bf403220" containerName="revoke-keys" Nov 01 22:41:29.892017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:29.892009 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fa8f34-4795-4188-835a-08d8bf403220" containerName="revoke-keys" Nov 01 22:41:29.968214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:29.892425 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6fa8f34-4795-4188-835a-08d8bf403220" containerName="revoke-keys" Nov 01 22:41:30.307709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.307680 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9"] Nov 01 22:41:30.307897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.307725 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z"] Nov 01 22:41:30.307897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.307747 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e3c26a-lvp2z"] Nov 01 22:41:30.307897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.307813 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.413105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.413073 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.413285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.413256 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f994ce54-fa92-4360-8e3a-4e332218e556-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.413370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.413357 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.413456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.413439 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f994ce54-fa92-4360-8e3a-4e332218e556-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.413516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.413501 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.413684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.413670 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcj7q\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-kube-api-access-bcj7q\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.413732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.413699 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.413780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.413728 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.413821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.413792 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.517051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.516996 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.519546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.519345 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcj7q\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-kube-api-access-bcj7q\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.520936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.520903 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.522636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.522612 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.523126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.523092 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.524520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.523849 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.525037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.525019 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f994ce54-fa92-4360-8e3a-4e332218e556-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.526465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.523653 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.526637 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.524929 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.526747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.521495 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.539486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.525773 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f994ce54-fa92-4360-8e3a-4e332218e556-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.539486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.526909 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.539486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.526941 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f994ce54-fa92-4360-8e3a-4e332218e556-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.539486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.532319 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.729478 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.729397 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.729656 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.729630 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcj7q\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-kube-api-access-bcj7q\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.729727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.729676 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.730222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.730197 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f994ce54-fa92-4360-8e3a-4e332218e556-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:30.818108 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:30.818071 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:31.476594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:31.476562 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04aa83a3-acba-4a30-b195-f898e52d3825" path="/var/lib/kubelet/pods/04aa83a3-acba-4a30-b195-f898e52d3825/volumes" Nov 01 22:41:32.344629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:32.344586 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9"] Nov 01 22:41:32.587964 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:41:32.587931 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf994ce54_fa92_4360_8e3a_4e332218e556.slice/crio-9ac409c18d63fa03fab1e3abc2b7a32eaf68702ee8f7e220fc45d89729f0c79c WatchSource:0}: Error finding container 9ac409c18d63fa03fab1e3abc2b7a32eaf68702ee8f7e220fc45d89729f0c79c: Status 404 returned error can't find the container with id 9ac409c18d63fa03fab1e3abc2b7a32eaf68702ee8f7e220fc45d89729f0c79c Nov 01 22:41:32.592821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:32.592784 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:32.592936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:32.592882 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:32.593002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:32.592932 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:33.071767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:33.071730 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" event={"ID":"f994ce54-fa92-4360-8e3a-4e332218e556","Type":"ContainerStarted","Data":"a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c"} Nov 01 22:41:33.071982 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:33.071793 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" event={"ID":"f994ce54-fa92-4360-8e3a-4e332218e556","Type":"ContainerStarted","Data":"9ac409c18d63fa03fab1e3abc2b7a32eaf68702ee8f7e220fc45d89729f0c79c"} Nov 01 22:41:33.097695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:33.097642 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podStartSLOduration=4.09762696 podStartE2EDuration="4.09762696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:33.094557513 +0000 UTC m=+1105.743056759" watchObservedRunningTime="2026-11-01 22:41:33.09762696 +0000 UTC m=+1105.746126188" Nov 01 22:41:33.110389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:33.110251 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:33.110389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:33.110352 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:33.818882 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:33.818837 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:33.820572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:33.820539 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:41:33.820729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:33.820590 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:41:34.039486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:34.039449 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg"] Nov 01 22:41:34.056448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:34.056403 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa36466-9qdhg"] Nov 01 22:41:34.818921 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:34.818883 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:41:34.819343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:34.818941 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:41:35.505958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:35.505904 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c7006c-c170-4188-a7b5-27bfbbb6c82a" path="/var/lib/kubelet/pods/b3c7006c-c170-4188-a7b5-27bfbbb6c82a/volumes" Nov 01 22:41:35.818738 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:35.818650 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:41:35.818919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:35.818732 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:41:36.033183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:36.033142 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd"] Nov 01 22:41:36.042096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:36.042067 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9cc6c56-nv5vd"] Nov 01 22:41:36.819325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:36.819286 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:41:36.819523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:36.819360 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:41:37.577210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:37.572468 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078d57ed-1850-4485-b27f-b38514f5325a" path="/var/lib/kubelet/pods/078d57ed-1850-4485-b27f-b38514f5325a/volumes" Nov 01 22:41:37.818600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:37.818559 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:41:37.847903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:37.818691 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:41:38.693140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.673612 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:41:38.693140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.674012 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" podUID="a7025b36-aa82-40c1-9fc1-e1e4470498c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 01 22:41:38.793296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.755627 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq"] Nov 01 22:41:38.827137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.774269 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:41:38.827137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.774313 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:41:38.827137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.774346 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:41:38.909524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.878546 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": context deadline exceeded" Nov 01 22:41:38.909524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.886022 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:38.909524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.891990 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:41:38.909524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.893378 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 22:41:38.909524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.893432 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 22:41:38.909524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:38.893609 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:41:39.392667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.060483 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.392667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.061944 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.392667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.066229 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.392667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.066426 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.392667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.192245 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.392667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.268321 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.392667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.316742 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.392667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.363211 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.886124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.588253 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.886124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.674384 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:41:39.886124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.780218 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" podUID="a7025b36-aa82-40c1-9fc1-e1e4470498c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 01 22:41:39.886124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.809644 6287 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-xqjq8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.12:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:39.886124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.827220 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:39.886124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.827240 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:40.028568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.887046 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:40.028568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.893173 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:41:40.028568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.894251 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:41:40.028568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.898562 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:40.205911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.061413 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:40.205911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.066548 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:40.205911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.066548 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:40.205911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.066564 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:40.382987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.268991 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:40.675147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.675070 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:41:40.784927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.784561 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" podUID="a7025b36-aa82-40c1-9fc1-e1e4470498c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 01 22:41:40.820156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.819885 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:40.820156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.819975 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:40.892956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.828322 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:41:40.892956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.828366 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:42.066185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:42.066062 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:42.066185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:42.066174 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:42.206670 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:42.066396 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:42.206670 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:42.068878 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:42.468668 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:42.468586 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:41:42.596270 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:42.468828 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:41:42.596270 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:42.469063 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:42.596270 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:40.192815 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:41:43.605130 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:43.605058 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:43.605472 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:43.607967 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:43.752759 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:43.964042 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:43.964245 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.212218 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.269309 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" podUID="a7025b36-aa82-40c1-9fc1-e1e4470498c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.498316 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.498384 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.498413 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.498461 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.498487 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.638946 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.643915 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.648262 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.668219 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.875654 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" podUID="85879320-4b31-4f5e-a902-797e6f510711" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.50:50052\" within 1s: context deadline exceeded" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:41:44.875872 6287 logging.go:55] [core] [Channel #2556 SubChannel #2558]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.50:50052", ServerName: "10.133.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.50:50052: operation was canceled" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.897191 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.897264 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.900749 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.900789 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.942535 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:44.942632 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.034497 6287 patch_prober.go:29] interesting pod/jobset-controller-manager-76ffc984cf-cpvzx container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.034555 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" podUID="3633d07f-afe4-4477-bc86-d1881c6db757" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.055985 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.120643 6287 trace.go:236] Trace[1156375513]: "Calculate volume metrics of istio-podinfo for pod openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" (01-Nov-2026 22:41:39.608) (total time: 5511ms): Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: Trace[1156375513]: [5.511982399s] [5.511982399s] END Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:39.809721 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" podUID="7059b374-3864-4296-bd7b-d77161dfca28" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.12:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.126502 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.73627811s: [/var/lib/containers/storage/overlay/2950d303bd3aa4a0245989ef7dd65664ca469087a28f66e98811fc1e6de37044/diff /var/log/pods/openshift-ingress_e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b_c68acbcb-66c9-4def-af31-184d7544248d/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.131035 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.131078 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.147169 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.754895182s: [/var/lib/containers/storage/overlay/4698ce9f62bb38f780393bf6aa5f7df5a33ce419f716de03692c059c6fbf01c7/diff /var/log/pods/e2e-models-e2e-worker-w0-e25182_distinct-w0-e25182-kserve-96f66848b-b5d2h_5bc29f3d-8367-4958-bb4e-a489d5ab620e/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.250313 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2"] Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.250526 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.351570 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.351618 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.351645 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.351810 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b38708-4a75-40df-9835-d15d80301461-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.351860 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tcbb\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-kube-api-access-2tcbb\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.351989 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b38708-4a75-40df-9835-d15d80301461-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.352098 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.352183 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.352261 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.382232 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq"] Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.382285 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2"] Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.382350 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.386749 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.386753 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-425870c1-serving-cert\"" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.386806 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7x7m\"" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.386812 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.386892 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.423302 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk"] Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.460503 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.460557 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.460583 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.460768 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b38708-4a75-40df-9835-d15d80301461-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.460849 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tcbb\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-kube-api-access-2tcbb\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.460934 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-metrics-certs\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.460963 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b38708-4a75-40df-9835-d15d80301461-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.461186 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.461215 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-maas-api-tls\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.461248 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm655\" (UniqueName: \"kubernetes.io/projected/2caef5a6-ee83-4304-b8be-583b397fb350-kube-api-access-rm655\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.461449 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caef5a6-ee83-4304-b8be-583b397fb350-openshift-service-ca\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.461497 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.462044 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.564451 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-metrics-certs\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.564626 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-maas-api-tls\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.564663 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm655\" (UniqueName: \"kubernetes.io/projected/2caef5a6-ee83-4304-b8be-583b397fb350-kube-api-access-rm655\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.564762 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caef5a6-ee83-4304-b8be-583b397fb350-openshift-service-ca\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.802158 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.805614 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3e228e-serving-cert\"" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.812785 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk"] Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.812873 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.812890 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4"] Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.870138 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f064454-385d-4336-abb8-288d642fe6fc-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.870461 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-metrics-certs\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.870639 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mq5\" (UniqueName: \"kubernetes.io/projected/0f064454-385d-4336-abb8-288d642fe6fc-kube-api-access-r7mq5\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.870781 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-maas-api-tls\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.921028 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.921079 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.967013 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" podUID="c74aadae-e2f9-47f6-b533-88919cce7c90" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.972518 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mq5\" (UniqueName: \"kubernetes.io/projected/0f064454-385d-4336-abb8-288d642fe6fc-kube-api-access-r7mq5\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.972658 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-maas-api-tls\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.972800 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f064454-385d-4336-abb8-288d642fe6fc-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:45.973053 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-metrics-certs\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.164213 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.164256 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.213245 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.213301 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.356770 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.473411 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.473468 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.534718 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b38708-4a75-40df-9835-d15d80301461-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.738972 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.739044 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x"] Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.739085 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.791891 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rb2x\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-kube-api-access-5rb2x\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.791958 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.792148 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.792267 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.792323 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.792450 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.792522 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.792557 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.792666 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.902373 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.902531 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.902591 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.902709 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.902796 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.902866 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.902996 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.903054 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rb2x\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-kube-api-access-5rb2x\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.913654 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.963223 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:46.963315 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.069411 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.069468 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.307723 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.308039 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm655\" (UniqueName: \"kubernetes.io/projected/2caef5a6-ee83-4304-b8be-583b397fb350-kube-api-access-rm655\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.361193 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-maas-api-tls\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.361289 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-maas-api-tls\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.487603 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-metrics-certs\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.529303 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caef5a6-ee83-4304-b8be-583b397fb350-openshift-service-ca\") pod \"maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.564218 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.596811 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tcbb\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-kube-api-access-2tcbb\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.683375 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f064454-385d-4336-abb8-288d642fe6fc-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.683435 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.683577 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mq5\" (UniqueName: \"kubernetes.io/projected/0f064454-385d-4336-abb8-288d642fe6fc-kube-api-access-r7mq5\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.744826 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.777993 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.778050 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94"] Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.815042 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b38708-4a75-40df-9835-d15d80301461-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.815602 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-metrics-certs\") pod \"maas-api-e2e-api-a-3e228e-655579484b-ftswk\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.824806 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.825094 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:47.899370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.825179 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.825290 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.825349 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.825529 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.825621 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rb2x\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-kube-api-access-5rb2x\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.825748 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.825779 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.853534 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.853755 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.853844 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.854009 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8clxn\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-kube-api-access-8clxn\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.854088 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.854193 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.854224 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.854269 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.854313 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.956037 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.956152 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.956175 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.956204 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.956228 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.956272 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.956419 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.956469 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.956554 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8clxn\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-kube-api-access-8clxn\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.959218 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.959259 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.974049 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.981836 6287 generic.go:356] "Generic (PLEG): container finished" podID="dd434389-e042-41e4-a84f-325058069d49" containerID="d6db61681bcf311ee0144282ca573441c65c9a49195ad1757bb7cb832fff93c4" exitCode=0 Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.982005 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:47.982423 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.238483 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.238490 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4"] Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.238610 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.238713 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.238762 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.238946 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.238968 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.239008 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs"] Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.366604 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.366675 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff2728-08a8-4ebe-93bf-e240c7639d50-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.366711 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.366763 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.366842 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q886l\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-kube-api-access-q886l\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.366878 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.367341 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.367493 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.367595 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.372768 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.373493 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.374690 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8clxn\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-kube-api-access-8clxn\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.376194 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.376304 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.472462 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.472513 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff2728-08a8-4ebe-93bf-e240c7639d50-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.472557 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.472672 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.472748 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q886l\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-kube-api-access-q886l\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.472777 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.473006 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.473477 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.473572 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.523564 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.524282 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.558146 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.558256 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.558292 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.558566 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94"] Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.558609 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" event={"ID":"dd434389-e042-41e4-a84f-325058069d49","Type":"ContainerDied","Data":"d6db61681bcf311ee0144282ca573441c65c9a49195ad1757bb7cb832fff93c4"} Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.558690 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.558743 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.558758 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q886l\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-kube-api-access-q886l\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.558791 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs"] Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.559039 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.559170 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.559466 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.559722 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.559755 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x"] Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560236 6287 scope.go:122] "RemoveContainer" containerID="d6db61681bcf311ee0144282ca573441c65c9a49195ad1757bb7cb832fff93c4" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560572 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560616 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560672 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560715 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560755 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560792 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560820 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560859 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560875 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560906 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560934 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560952 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.560980 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.561007 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.561035 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.561060 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.561089 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.561133 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.561161 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.561193 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.561219 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.576570 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.664042 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.664265 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff2728-08a8-4ebe-93bf-e240c7639d50-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.664298 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.664340 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.694447 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.694793 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.694865 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8gg6\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-kube-api-access-x8gg6\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.695020 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.695053 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.695086 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.695145 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.695228 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.695452 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:41:48.700665 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.796702 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.796743 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.796769 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.796805 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.796858 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.797035 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.797159 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.797219 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.797257 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8gg6\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-kube-api-access-x8gg6\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.928961 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-data\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.929351 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istiod-ca-cert\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.929508 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-envoy\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.929632 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.929678 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-credential-socket\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.929820 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-certs\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.930057 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-podinfo\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.932022 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-token\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.932743 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8gg6\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-kube-api-access-x8gg6\") pod \"e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:48.953710 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:49.079001 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:49.079091 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:49.079142 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:49.309851 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:50.051649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:49.534003 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:41:51.410999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:51.408980 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" event={"ID":"dd434389-e042-41e4-a84f-325058069d49","Type":"ContainerStarted","Data":"b665e428280f3684fcf032d352586a1f2126a289089417df448d1c6d1b555a5f"} Nov 01 22:41:51.768832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:51.768746 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:41:52.770791 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:52.770660 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:41:52.770791 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:52.770719 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:41:53.040136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:53.039220 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:41:53.205228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:53.190304 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:53.831139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:53.817252 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:41:53.831139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:53.817314 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:41:53.868927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:53.868880 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:54.257267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.257211 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:54.282653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.258618 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:41:54.292103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.290124 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:41:54.292103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.265845 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:41:54.350494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.350442 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2"] Nov 01 22:41:54.378904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.378814 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk"] Nov 01 22:41:54.406775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.406692 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x"] Nov 01 22:41:54.426477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.425606 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:54.792383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.754096 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:54.792383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.754492 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:41:54.792383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.757403 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:41:54.792383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.757505 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:41:54.974232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.925717 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:41:55.035354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:54.993856 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:41:55.108293 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.092238 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:55.170236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.170188 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:41:55.170747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.170717 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:41:55.305283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.260686 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:55.468966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.468867 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:41:55.491549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.491506 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:41:55.499516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.499478 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Nov 01 22:41:55.554141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.552169 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:41:55.572611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.572582 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:41:55.573095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.573078 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:41:55.579252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.579219 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:55.602159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.599961 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:41:55.602159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.600030 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:41:55.614482 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.607093 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:55.614482 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.607454 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:41:55.783045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.782994 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:41:55.838144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.804956 6287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:41:56.024724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:55.805076 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:56.024724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.008607 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:56.024724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.023141 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:41:56.024724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.023203 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:41:56.234200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.232215 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:56.351161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.350964 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:41:56.594345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.594291 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:56.903262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.599532 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.903262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.633201 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.917308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.652545 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:56.917475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.653923 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:41:56.920651 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:41:56.580983 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f064454_385d_4336_abb8_288d642fe6fc.slice/crio-34376cff6258d858c67bbc9a64cce952b01c5965fe13ef12898c623fd11b0532 WatchSource:0}: Error finding container 34376cff6258d858c67bbc9a64cce952b01c5965fe13ef12898c623fd11b0532: Status 404 returned error can't find the container with id 34376cff6258d858c67bbc9a64cce952b01c5965fe13ef12898c623fd11b0532 Nov 01 22:41:56.920651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.918307 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.920651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.809501 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:41:56.920651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.918639 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.920651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.878613 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:56.920651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.918734 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.920651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.918891 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.923706 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.922847 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:41:56.923706 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.829855 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:41:56.950169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.754255 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.950169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.949438 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.950169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.949517 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.970193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.723208 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.970193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.957484 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.970193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.957546 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:56.970193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.755717 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:41:57.032151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.031264 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:57.032151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.851046 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:41:57.032700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.871563 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:57.032700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.032300 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:57.032700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.032360 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:41:57.033037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.769066 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:57.033348 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.755773 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:57.033924 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:56.835182 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:57.054138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.053711 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:41:57.062720 6287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.352s" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.062816 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.062832 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94"] Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.062889 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.062904 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs"] Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.062925 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq"] Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.062962 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.062984 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.062995 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4"] Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.063023 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.063045 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.063065 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.063087 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.063123 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:41:57.070096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.068064 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:41:57.136459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.136344 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:41:57.142760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.140504 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:41:57.143014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.142994 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:41:57.144394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.144363 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:41:57.144559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.144545 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:41:57.242415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.242373 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:41:57.249268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.249242 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:41:57.249666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.249645 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:41:57.261617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.250809 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:41:57.261617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.260368 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" event={"ID":"0d3bf83f-b364-4622-975c-f79c1f7cd3c0","Type":"ContainerStarted","Data":"73a9b7bea0746b84c0991035ec261d1c4752d7859a5fb0f95ee021f6e4972f5a"} Nov 01 22:41:57.268754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.263085 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:41:57.364705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.364327 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" event={"ID":"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe","Type":"ContainerStarted","Data":"8c735938d8caa258d928b95b9dbf6b57255d05b4da082916429bf265180ba4f9"} Nov 01 22:41:57.368649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.368619 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" event={"ID":"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa","Type":"ContainerStarted","Data":"1d544c3c43a7c1b201e394b88df70b2271d0654febf22576c79c355bc034e78a"} Nov 01 22:41:57.431910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.431816 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" event={"ID":"fdff2728-08a8-4ebe-93bf-e240c7639d50","Type":"ContainerStarted","Data":"ef9aaa8e4d2152bd0d49e54041b6d960c79288c6230befb7309a18a3d872f738"} Nov 01 22:41:57.435740 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.435710 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" event={"ID":"2caef5a6-ee83-4304-b8be-583b397fb350","Type":"ContainerStarted","Data":"36f47546981a0cff61e39e7e75ec289bd1c3e82d354b5f5637b2412fa1a17a0d"} Nov 01 22:41:57.438356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.438184 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" event={"ID":"c8b38708-4a75-40df-9835-d15d80301461","Type":"ContainerStarted","Data":"213c3061742161b125806e59d7528ad96c8c190e7474a3c059816b77255560a7"} Nov 01 22:41:57.544098 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:57.544050 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" event={"ID":"0f064454-385d-4336-abb8-288d642fe6fc","Type":"ContainerStarted","Data":"34376cff6258d858c67bbc9a64cce952b01c5965fe13ef12898c623fd11b0532"} Nov 01 22:41:58.034102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:58.034072 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:41:58.035399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:58.034718 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:41:58.923202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:58.902992 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:41:58.923202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:58.919819 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:41:59.362192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:59.331649 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:41:59.463561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:59.463498 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podStartSLOduration=15.463478513 podStartE2EDuration="15.463478513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:59.445423173 +0000 UTC m=+1132.093922410" watchObservedRunningTime="2026-11-01 22:41:59.463478513 +0000 UTC m=+1132.111977749" Nov 01 22:41:59.538721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:59.538395 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podStartSLOduration=15.538374190999999 podStartE2EDuration="15.538374191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:59.487152359 +0000 UTC m=+1132.135651595" watchObservedRunningTime="2026-11-01 22:41:59.538374191 +0000 UTC m=+1132.186873430" Nov 01 22:41:59.765578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:59.765005 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:41:59.765578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:59.765087 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:41:59.965716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:41:59.964348 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:42:00.314794 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.314746 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" event={"ID":"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe","Type":"ContainerStarted","Data":"1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f"} Nov 01 22:42:00.319008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.318974 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:00.319246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.319027 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:00.321741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.321714 6287 generic.go:356] "Generic (PLEG): container finished" podID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerID="f9c509da9a833d89d77807d307cb416dee4072583a498238bb025c08028e4676" exitCode=0 Nov 01 22:42:00.322014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.321995 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" event={"ID":"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186","Type":"ContainerDied","Data":"f9c509da9a833d89d77807d307cb416dee4072583a498238bb025c08028e4676"} Nov 01 22:42:00.322678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.322659 6287 scope.go:122] "RemoveContainer" containerID="f9c509da9a833d89d77807d307cb416dee4072583a498238bb025c08028e4676" Nov 01 22:42:00.330892 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.330844 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:42:00.332700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.332631 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:42:00.333322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.331712 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" event={"ID":"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa","Type":"ContainerStarted","Data":"3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e"} Nov 01 22:42:00.333564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.333538 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:42:00.344102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.344078 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" event={"ID":"fdff2728-08a8-4ebe-93bf-e240c7639d50","Type":"ContainerStarted","Data":"53d5a8711670c52bcb680f0f13dc3f936f8d7414d17a1766054f0c9bc35f3c52"} Nov 01 22:42:00.348902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.348868 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" event={"ID":"c8b38708-4a75-40df-9835-d15d80301461","Type":"ContainerStarted","Data":"15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58"} Nov 01 22:42:00.349160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.349089 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podStartSLOduration=16.349071605 podStartE2EDuration="16.349071605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:00.339988815 +0000 UTC m=+1132.988488065" watchObservedRunningTime="2026-11-01 22:42:00.349071605 +0000 UTC m=+1132.997570841" Nov 01 22:42:00.353523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.353498 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" event={"ID":"0d3bf83f-b364-4622-975c-f79c1f7cd3c0","Type":"ContainerStarted","Data":"9f7e2056f648d663d49db9b855050a3c04096b5fb56fe55daccd38f223f6caff"} Nov 01 22:42:00.400592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.400533 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podStartSLOduration=23.40051312 podStartE2EDuration="23.40051312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:00.385940504 +0000 UTC m=+1133.034439739" watchObservedRunningTime="2026-11-01 22:42:00.40051312 +0000 UTC m=+1133.049012356" Nov 01 22:42:00.553160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.541766 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podStartSLOduration=16.54174541 podStartE2EDuration="16.54174541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:00.409608365 +0000 UTC m=+1133.058107605" watchObservedRunningTime="2026-11-01 22:42:00.54174541 +0000 UTC m=+1133.190244647" Nov 01 22:42:00.647760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.647669 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:42:00.671029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.671004 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:42:00.671327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.670263 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:00.671473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.671451 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:00.704141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.703314 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:00.704141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.703377 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:00.882540 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.882510 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:42:00.918579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.896697 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:42:00.985683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:00.982243 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:42:01.089962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:01.089926 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:01.090287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:01.090242 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:01.124985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:01.090475 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:01.125460 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:01.125332 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:01.225194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:01.224749 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:42:01.984380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:01.974947 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:01.984380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:01.979061 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:01.984380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:01.979107 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:02.086168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.066323 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:42:02.124145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.109921 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:42:02.158669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.155288 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:02.158977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.158939 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:02.159132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.155910 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:42:02.159967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.155929 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:42:02.159967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.157624 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:02.159967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.155861 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:02.159967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.155342 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:02.159967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.159911 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:02.240166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.240132 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:02.240487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.240437 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:02.240735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.240718 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:02.240885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.240866 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:02.302018 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.301974 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:02.327709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.321754 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:02.327709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.321806 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:02.428508 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.399718 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:02.635271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.575501 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:02.635271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.575555 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:02.635271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.611845 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" event={"ID":"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186","Type":"ContainerStarted","Data":"66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667"} Nov 01 22:42:02.635271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.630825 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:02.635271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.630887 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:02.675371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.658937 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:02.744141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.724204 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:02.944521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.924910 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:42:03.119136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.944827 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:03.119136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.964684 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:03.119136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.115019 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:03.119136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.992218 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:42:03.167786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.992077 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:42:03.170321 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.170283 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:42:03.185929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:02.992154 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:03.211358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.211263 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:03.211590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.187934 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:42:03.212455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.010301 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:03.212974 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.187881 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:03.230315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.187955 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:42:03.246307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.231012 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:42:03.246684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.246651 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:42:03.306517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.187984 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:42:03.334963 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.188006 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:42:03.360861 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.319574 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:42:03.372054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.361105 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:42:03.372054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.361189 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:42:03.372054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.321233 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:42:03.372054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.360448 6287 trace.go:236] Trace[1495474441]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" (01-Nov-2026 22:42:02.001) (total time: 1134ms): Nov 01 22:42:03.372054 ip-10-0-128-55 kubenswrapper[6287]: Trace[1495474441]: [1.134050641s] [1.134050641s] END Nov 01 22:42:03.372054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.363243 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:42:03.432147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.427855 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:42:03.432147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.428525 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:42:03.685187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.502642 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:42:03.685187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.679644 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:42:03.738180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.737263 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:03.738180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.737347 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:03.738180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.737951 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:03.738180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.737989 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:03.902153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.898153 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:03.906803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.906753 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:04.133814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.920812 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:04.238819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.238784 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:04.239108 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.130319 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:04.284644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.924672 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:04.284644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.970594 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:04.284644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.275179 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:04.284644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:03.980847 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:04.284644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.275250 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:04.284644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.164333 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:42:04.322582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.164592 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:04.322582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.237978 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:04.339456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.339396 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:04.393594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.368634 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:42:04.596638 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.441320 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:42:04.597262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.441472 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:42:04.597919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.441516 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:42:04.622606 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.610402 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" podUID="a7025b36-aa82-40c1-9fc1-e1e4470498c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 01 22:42:04.622606 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.612743 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 01 22:42:04.622606 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.612838 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:42:04.667351 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.632293 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:04.667351 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.658667 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:04.667351 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.658746 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:04.694212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.667831 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:42:04.694212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.690290 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:04.694212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.690405 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:42:04.695763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.694624 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Nov 01 22:42:04.994201 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:42:04.955936 6287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.392s" Nov 01 22:42:05.020139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.964243 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:05.020139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.008357 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:05.020139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:04.967271 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:05.020139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.008486 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:05.111152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.111091 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:42:05.158146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.152464 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.309413 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.309536 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.309720 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.309760 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.309797 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.309854 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.309906 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.309924 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.134601 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.309992 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:05.361227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.308758 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:05.362092 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.362047 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:05.451169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.363673 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:05.600937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.174183 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:05.600937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.588656 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:05.600937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.551867 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:42:05.642464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.551886 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:42:05.713292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.551904 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:42:05.713292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.551801 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:42:05.713292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.676272 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:05.713292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.551840 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:05.713292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.691609 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:05.713292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.691656 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:05.713292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.692173 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:05.713292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.692204 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:05.731167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.551961 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": context deadline exceeded" Nov 01 22:42:05.731167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.579076 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:05.731167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.716410 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:05.731167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.716460 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:05.782739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.733090 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:42:05.782739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.733412 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:42:05.782739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.733625 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:42:05.782739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.739489 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:05.782739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.739680 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:42:05.782739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.739828 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:42:05.782739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.763597 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:05.782739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.763653 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:05.990312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.990278 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:06.012443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.001194 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:06.012775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:05.994042 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:06.026099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.012875 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:06.265239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.120778 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:42:06.383736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.383693 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:06.569191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.492848 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:06.569191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.542754 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:06.569191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.561295 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:06.569191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.561351 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:06.569191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.561524 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:06.569191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.561551 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:06.602170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.572163 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:06.602170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.572221 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:06.657457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.657410 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:06.724027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.682946 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:06.724463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.682981 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:42:06.724739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.712597 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:06.813150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.712631 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:42:06.836198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.763298 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:06.890356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.787539 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:42:06.909449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.787572 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:42:07.077382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.787593 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:42:07.077382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.787632 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.077382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.008388 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.077382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.787668 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:42:07.077382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:06.787650 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:42:07.077382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.030453 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:07.077382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.030507 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:07.077382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.059228 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:07.077382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.059288 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:07.355645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.355606 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:07.381276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.381225 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:07.385588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.385549 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.498235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.451464 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:07.498235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.451529 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:07.498235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.451814 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.539152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.538349 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:42:07.573995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.573960 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:07.629368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.629270 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:07.749514 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.749467 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:07.749897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.749850 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:07.813136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.787903 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.827716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.793774 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:42:07.852141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.793810 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:07.854198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.805734 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.863136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.827156 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.892515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.892400 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.912511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.912276 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.947390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.947335 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:07.947669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.947626 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:07.983590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:07.983557 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n"] Nov 01 22:42:08.017089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.013471 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:08.017089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.013903 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:08.017089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.014238 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:08.017089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.014502 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:42:08.017089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.014771 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:08.077373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.077328 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:08.178234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.174605 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:08.208337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.110909 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:08.222168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.079766 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:08.222450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.222417 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:08.270264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.270213 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:08.334574 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.334548 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:42:08.336628 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.336603 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:42:08.337279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.337252 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:08.362279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.337580 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:08.386380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.386337 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:08.459805 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.459696 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2kx\" (UniqueName: \"kubernetes.io/projected/31de062f-1c3b-4e81-b8e2-df7807b30cd9-kube-api-access-hq2kx\") pod \"maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:08.460463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.460263 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:08.460716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.460696 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:08.464144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.461589 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:08.464144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.461639 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:08.562435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.562396 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:08.562708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.562679 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:08.564488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.564462 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:08.568214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.568177 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:08.573193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.565437 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:08.587448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.565394 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:08.587448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.582316 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2kx\" (UniqueName: \"kubernetes.io/projected/31de062f-1c3b-4e81-b8e2-df7807b30cd9-kube-api-access-hq2kx\") pod \"maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:08.587448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.582574 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:08.587448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.582704 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:08.828464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.828402 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:42:08.829612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.829581 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:42:08.870173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.856262 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:08.870173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.860017 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:42:08.893286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.882586 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": context deadline exceeded" Nov 01 22:42:08.893286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.882944 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:42:08.925855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.925810 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:42:08.972691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:08.972637 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:42:09.076160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.074921 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:09.076160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.075742 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:42:09.080209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.076727 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:42:09.146137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.145347 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:42:09.182139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.165091 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:42:09.237138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.230185 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:09.283560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.283512 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:09.284077 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.284027 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:09.288134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.284729 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:09.288335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.288310 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:09.410914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.410799 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:09.562137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.536822 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:09.562137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.440462 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:09.562137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.440021 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:09.588267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.479222 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:09.588523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.588490 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:09.589224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.462674 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded" Nov 01 22:42:09.592198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.591498 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:42:09.645675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.645636 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:09.646006 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.645962 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:09.694625 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.678811 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2kx\" (UniqueName: \"kubernetes.io/projected/31de062f-1c3b-4e81-b8e2-df7807b30cd9-kube-api-access-hq2kx\") pod \"maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:09.718146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.695281 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:09.862191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.830725 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:10.022247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.022200 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:10.022515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.956077 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:10.023022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.956008 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:10.023579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.004003 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:10.023579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.023475 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:10.047455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.852677 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:10.051191 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.956101 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:42:10.051736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.956156 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:42:10.064197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.956178 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:42:10.064778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:09.985533 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:42:10.087609 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.087562 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:42:10.088333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.088300 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:42:10.116902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.116853 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:10.158139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.134989 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:10.158502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.158463 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:10.164129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.164080 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:42:10.273217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.271857 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:10.273810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.273628 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:42:10.364455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.364421 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n"] Nov 01 22:42:10.365010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.364986 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:10.388375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.388314 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:10.449311 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.449185 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:10.450123 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.449987 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:10.644207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.609455 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:10.691140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.690455 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:10.691140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.690514 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:10.691140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.690604 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:42:10.769141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.736992 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 22:42:10.833731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.833688 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 22:42:10.848485 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.807211 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:42:10.976690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:10.960148 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:11.183279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.183237 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 01 22:42:11.183561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.183529 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 01 22:42:11.284755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.284706 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:11.350139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.338244 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:11.402516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.360675 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:11.434136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.360724 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:42:11.487139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.360754 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:11.559338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.559245 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:11.624477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.360795 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:42:11.667210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.666036 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded" Nov 01 22:42:11.667210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.360832 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:42:11.668553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.360871 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:42:11.668731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.365390 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.365424 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.522670 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.676317 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.360775 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.523128 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.525749 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.677895 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.556587 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.677942 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.586250 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.677982 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:11.692136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.612184 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:42:11.716937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.360813 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:42:12.003186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:11.990062 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:42:12.283193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.126525 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 01 22:42:12.301431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.301379 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 01 22:42:12.435296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.435258 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:12.436285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.436253 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:42:12.533178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.435207 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:12.533178 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:42:12.492451 6287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.021s" Nov 01 22:42:12.533178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.492564 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:42:12.533178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.493145 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:42:12.533178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.517309 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:42:12.598258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.559342 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:12.598258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.559403 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:12.617146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.616451 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:12.631333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.621689 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:42:12.642040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.631842 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:12.660417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.658016 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 22:42:12.660417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.658078 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 22:42:12.743137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.666553 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:42:12.743137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.723514 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:42:12.743137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.742132 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:42:12.762088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.762048 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:13.005193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.862739 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:42:13.005193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.862777 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:42:13.005193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:13.004917 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:42:13.005193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.862799 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:42:13.005530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.862839 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:42:13.005530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.931911 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:13.005643 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:12.862819 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:42:13.464395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:13.454709 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:42:13.464395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:13.459983 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:13.520950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:13.520910 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:13.539529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:13.539499 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:42:13.550912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:13.550871 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 22:42:13.557170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:13.555900 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:13.561356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:13.561291 6287 scope.go:122] "RemoveContainer" containerID="ed382ced0491387b641688e778629b34f86c2a6a432b47b4bf19184fce8cb3b9" Nov 01 22:42:13.668354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:13.646293 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:42:14.029490 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.029409 6287 scope.go:122] "RemoveContainer" containerID="a2110b3435b241660485d4b002a6615ea2fcea030d1e5a1cb52e7c25453c2ec5" Nov 01 22:42:14.061628 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.061585 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:14.089137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.080325 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:42:14.103133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.092374 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:42:14.112386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.109399 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:42:14.112386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.109497 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:42:14.112386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.109556 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:42:14.112386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.109813 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:42:14.116099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.116076 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:42:14.117308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.117283 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925"] Nov 01 22:42:14.132875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.132853 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:42:14.133050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.133034 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925"] Nov 01 22:42:14.133240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.133227 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:42:14.133412 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.133400 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:42:14.133531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.133520 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:42:14.133645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.133636 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:42:14.133756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.133747 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:42:14.133861 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.133850 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:42:14.133997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.133986 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:42:14.134309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.134293 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:42:14.134528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.134510 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:42:14.143745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.143705 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9b7320-dockercfg-vkq5p\"" Nov 01 22:42:14.145520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.145480 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:42:14.175330 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.173231 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9b7320\"" Nov 01 22:42:14.204095 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.204010 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750b6f4-bd5e-4480-9789-153358e1335b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925\" (UID: \"5750b6f4-bd5e-4480-9789-153358e1335b\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:42:14.204526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.204503 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/5750b6f4-bd5e-4480-9789-153358e1335b-kube-api-access-ljltx\") pod \"payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925\" (UID: \"5750b6f4-bd5e-4480-9789-153358e1335b\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:42:14.230572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.230547 6287 scope.go:122] "RemoveContainer" containerID="9a6139b9ba598472c47e600be51ae75f22ace42b0bb8bdd21fe22e1388eb4ed5" Nov 01 22:42:14.294185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.294152 6287 scope.go:122] "RemoveContainer" containerID="a32336e110ecd0e514ff70331694699c44fae5e1e5a6af1eab64386e544e8374" Nov 01 22:42:14.307369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.307326 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750b6f4-bd5e-4480-9789-153358e1335b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925\" (UID: \"5750b6f4-bd5e-4480-9789-153358e1335b\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:42:14.308152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.308090 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/5750b6f4-bd5e-4480-9789-153358e1335b-kube-api-access-ljltx\") pod \"payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925\" (UID: \"5750b6f4-bd5e-4480-9789-153358e1335b\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:42:14.313257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.313231 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750b6f4-bd5e-4480-9789-153358e1335b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925\" (UID: \"5750b6f4-bd5e-4480-9789-153358e1335b\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:42:14.340880 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.340841 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n"] Nov 01 22:42:14.365727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.365650 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/5750b6f4-bd5e-4480-9789-153358e1335b-kube-api-access-ljltx\") pod \"payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925\" (UID: \"5750b6f4-bd5e-4480-9789-153358e1335b\") " pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:42:14.405085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.405056 6287 scope.go:122] "RemoveContainer" containerID="c126d130ce2c7b9d5e63445550210a465c22f830563eab1279144889c58e24e4" Nov 01 22:42:14.444883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.443318 6287 scope.go:122] "RemoveContainer" containerID="d8d33ec10b55b63c5949cdb7c898a1acadfdd60c5a6130f2201121a3371446d6" Nov 01 22:42:14.476383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.476187 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:42:14.526986 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.526919 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" event={"ID":"31de062f-1c3b-4e81-b8e2-df7807b30cd9","Type":"ContainerStarted","Data":"1f8dcfd3bb8bd9ca97856bde6f94153696d13774512399c7b4c34c65a523a59e"} Nov 01 22:42:14.535266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.534708 6287 scope.go:122] "RemoveContainer" containerID="2ec4117630bd23152dd1233eaaacd4fdb88befbc5e7964bb67df0fc63dac69e9" Nov 01 22:42:14.630185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.620827 6287 scope.go:122] "RemoveContainer" containerID="15bc607c4ed0a92fdd453cf3e007cd983195336fb801cade2d342ea04d8b1ec4" Nov 01 22:42:14.630185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.628002 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:42:14.631325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.631296 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:42:14.667952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.667927 6287 scope.go:122] "RemoveContainer" containerID="1ef86b6d9d1d695cac490876d3d7c11ad7f581dfb7523ea31a43eaecfa1d5972" Nov 01 22:42:14.724162 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.724074 6287 scope.go:122] "RemoveContainer" containerID="f12c0a2171b504f9a01db45c7defc65171e01d86ee66357cbb16459f30f9fa73" Nov 01 22:42:14.785667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.785643 6287 scope.go:122] "RemoveContainer" containerID="fdbd5d7d9a98d5e7ab16b4c7e19f7240823eb82b7d3cbd19b33229318f89265a" Nov 01 22:42:14.802382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.802354 6287 scope.go:122] "RemoveContainer" containerID="f5c6ebb8c103dfed5f3a47e9a7d0faaebc89e7c95150a3be9804742b5aafe576" Nov 01 22:42:14.955721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.955696 6287 scope.go:122] "RemoveContainer" containerID="51d40b55b4d1e0072304bfd74960ddad87a7b56bfcf06107a4e24236ecc9b454" Nov 01 22:42:14.979268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.979244 6287 scope.go:122] "RemoveContainer" containerID="a98031ebe6595b421379c662b92533dd5c612bab357095e549f9e63186c6c1b0" Nov 01 22:42:14.991053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.991024 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925"] Nov 01 22:42:14.991181 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:42:14.991086 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5750b6f4_bd5e_4480_9789_153358e1335b.slice/crio-feb6c85dde5e4e179e3461916e71e7a3487a7d09ab59dc8ee269f97f083545d4 WatchSource:0}: Error finding container feb6c85dde5e4e179e3461916e71e7a3487a7d09ab59dc8ee269f97f083545d4: Status 404 returned error can't find the container with id feb6c85dde5e4e179e3461916e71e7a3487a7d09ab59dc8ee269f97f083545d4 Nov 01 22:42:14.998986 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:14.998963 6287 scope.go:122] "RemoveContainer" containerID="e873a25a98f1f4af192f47c019b1b8f2616bafbb1f47b7c84a6cf06c98a53339" Nov 01 22:42:15.023645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:15.023625 6287 scope.go:122] "RemoveContainer" containerID="1f25f97376d8220221c602d3f0c5311319542a33678cc0835134d211bd7a8f46" Nov 01 22:42:15.047321 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:15.047290 6287 scope.go:122] "RemoveContainer" containerID="52b6ce2810b25aae610a8dc8174dda987f567a6e52c3057e738db7870142636a" Nov 01 22:42:15.696676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:15.696640 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" event={"ID":"5750b6f4-bd5e-4480-9789-153358e1335b","Type":"ContainerStarted","Data":"2db923b28461ce51515e59bd1627835d6e1598d2e043ff12ac73f04267d67bde"} Nov 01 22:42:15.697309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:15.697287 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" event={"ID":"5750b6f4-bd5e-4480-9789-153358e1335b","Type":"ContainerStarted","Data":"feb6c85dde5e4e179e3461916e71e7a3487a7d09ab59dc8ee269f97f083545d4"} Nov 01 22:42:15.699247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:15.699201 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:42:15.704506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:15.704482 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" event={"ID":"31de062f-1c3b-4e81-b8e2-df7807b30cd9","Type":"ContainerStarted","Data":"94bb13335fd5335e12a612b47d2a21a4c212dc3858006ea74a47c1b5a3925cd0"} Nov 01 22:42:15.737272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:15.737212 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" podStartSLOduration=4.737192491 podStartE2EDuration="4.737192491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:15.725506543 +0000 UTC m=+1148.374005769" watchObservedRunningTime="2026-11-01 22:42:15.737192491 +0000 UTC m=+1148.385691727" Nov 01 22:42:15.756551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:15.756319 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" podStartSLOduration=10.756300428 podStartE2EDuration="10.756300428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:15.749484705 +0000 UTC m=+1148.397983940" watchObservedRunningTime="2026-11-01 22:42:15.756300428 +0000 UTC m=+1148.404799670" Nov 01 22:42:15.929684 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:42:15.928623 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:42:16.717602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:16.717561 6287 generic.go:356] "Generic (PLEG): container finished" podID="31de062f-1c3b-4e81-b8e2-df7807b30cd9" containerID="94bb13335fd5335e12a612b47d2a21a4c212dc3858006ea74a47c1b5a3925cd0" exitCode=0 Nov 01 22:42:16.718081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:16.717635 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" event={"ID":"31de062f-1c3b-4e81-b8e2-df7807b30cd9","Type":"ContainerDied","Data":"94bb13335fd5335e12a612b47d2a21a4c212dc3858006ea74a47c1b5a3925cd0"} Nov 01 22:42:22.472662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:22.472608 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:23.118102 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:23.158281 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:23.186642 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:23.624870 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:23.625273 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:23.634067 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:23.669744 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:20.994341 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:23.669874 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:23.669918 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:24.219011 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": context deadline exceeded" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:24.219088 6287 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-xqjq8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.12:9443/health\": context deadline exceeded" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:24.231653 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:24.371684 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:24.371803 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:24.878321 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.255199 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-7sgls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.345624 6287 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bjq2j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.364290 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" podUID="230255df-b6dc-43d5-bcd7-364d2396ed06" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.55:50052: i/o timeout" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.676067 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.623318 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.51:50052\" within 1s: context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.345705 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" podUID="2957d06d-fec2-40f0-a4b4-b433ad6ac848" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.102:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.678438 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" podUID="bffc1a34-b42b-45e6-9350-31089c369391" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.678672 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.682000 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-vp7k9" podUID="2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.133.0.31:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.682078 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-bqwkz" podUID="d35d0387-f298-43a2-977a-51386c17077a" containerName="ovnkube-controller" probeResult="failure" output="command timed out" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.682285 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="97258e3b-22d2-487b-932c-d8dd9432d372" containerName="prometheus" probeResult="failure" output="command timed out" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733210 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733401 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733465 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733551 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733609 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733671 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Liveness probe status=failure output="Get \"https://10.133.0.15:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733694 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733732 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733746 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733796 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733848 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733862 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733935 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.733989 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734002 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734047 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734093 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734193 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734259 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734311 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734365 6287 patch_prober.go:29] interesting pod/dns-default-78mfv container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.8:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734378 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-78mfv" podUID="8c95cb60-d629-48ef-859d-85d7893afe81" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.8:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734433 6287 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9nqs9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.734452 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" podUID="f08dabd9-50ad-49a9-ba37-70a59b58c8c4" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.150:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.767406 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="97258e3b-22d2-487b-932c-d8dd9432d372" containerName="prometheus" probeResult="failure" output="command timed out" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:42.767516 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" podUID="a7025b36-aa82-40c1-9fc1-e1e4470498c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.041333 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.769182 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" podUID="a7025b36-aa82-40c1-9fc1-e1e4470498c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.769200 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.769245 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.769209 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.769513 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" podUID="bffc1a34-b42b-45e6-9350-31089c369391" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.772030 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.773193 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.773303 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.773489 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.773521 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.773566 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.773739 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.773870 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.773904 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.773768 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.774032 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.774149 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.774211 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.774240 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.774276 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.774457 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.775526 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.775717 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.776199 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.777418 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.741416701s: [/var/lib/containers/storage/overlay/dbcbf959bc22a9d2061683dae12d9ea5632034546848264e34136e58e8b39638/diff ]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.830867 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" podUID="85879320-4b31-4f5e-a902-797e6f510711" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.50:50052\" within 1s: context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.831806 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" podUID="7059b374-3864-4296-bd7b-d77161dfca28" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.12:9443/health\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.833976 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.162:50052\" within 1s: context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.836921 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.854018 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.466504516s: [/var/lib/containers/storage/overlay/53f66c76c2319217f296366be49fa6c9fd226cace68aea72c98b114ca4b87bbd/diff ]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.868379 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" podUID="230255df-b6dc-43d5-bcd7-364d2396ed06" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.872544 6287 patch_prober.go:29] interesting pod/jobset-controller-manager-76ffc984cf-cpvzx container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.872590 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" podUID="3633d07f-afe4-4477-bc86-d1881c6db757" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.873011 6287 patch_prober.go:29] interesting pod/jobset-controller-manager-76ffc984cf-cpvzx container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.873038 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" podUID="3633d07f-afe4-4477-bc86-d1881c6db757" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.928379 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" podUID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.53:50052\" within 1s: context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:42:43.932339 6287 logging.go:55] [core] [Channel #2826 SubChannel #2835]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.53:50052", ServerName: "10.133.0.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.53:50052: operation was canceled" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.958526 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" podUID="da7f0a84-0c8c-441e-8851-828d7d9949e8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:42:43.964358 6287 logging.go:55] [core] [Channel #2827 SubChannel #2878]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:50052", ServerName: "10.133.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.54:50052: operation was canceled" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967299 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.926406895s: [/var/lib/containers/storage/overlay/9c2ebed0a182bba1c3bb7a9a60e99173d53571292767f48d8983108d2d4c35ba/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/thanos-sidecar/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967331 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.926427225s: [/var/lib/containers/storage/overlay/cbfebfdfac4d965d12f0ec53a1c8a9f8f1133847f68617642c201b2b2952200a/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/kube-rbac-proxy-thanos/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967358 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.926453748s: [/var/lib/containers/storage/overlay/9ad1215ec5fbec58166a60ccc35e328b9e151eb60167b0e961106335aa6a3d82/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967378 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.930928723s: [/var/lib/containers/storage/overlay/85c781546bd632b78e61754fdd901b7d5a3442418c01e6336d335a64ab53b70b/diff /var/log/pods/e2e-models-e2e-worker-w5-aa441b_premium-w5-aa441b-kserve-66c7cbd7bb-m88cx_239a637b-db11-45f9-a13b-2739c43b99e5/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967399 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.818091147s: [/var/lib/containers/storage/overlay/ac23499f822ec5616120bf0a0725b63e1ad97e1cee847c1df487f9fa583ec8d1/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-87hds_71935876-95e7-4691-bedf-e040f4cfec12/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967420 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.57994319s: [/var/lib/containers/storage/overlay/bc8874f517ca44ae794ebd9d6ed0c3657d5800b8c1502497f07fac6b7d6072fd/diff /var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z26xx_bffc1a34-b42b-45e6-9350-31089c369391/manager/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967441 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.816791048s: [/var/lib/containers/storage/overlay/c9b9dd80cf55b97113f39ff4bef876dbae526c636cba323d0fd1e11397f17a82/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9nqs9_f08dabd9-50ad-49a9-ba37-70a59b58c8c4/discovery/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967461 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.131022229s: [/var/lib/containers/storage/overlay/17a74ff6f13da016b13740a1a427893c16f99ab1f13b6aacf10f4c70f25cef7b/diff /var/log/pods/opendatahub_ai-gateway-controller-6b6d674b9-llr7t_4459f450-7dab-4975-b2ac-02b042d1e004/manager/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967479 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.715110331s: [/var/lib/containers/storage/overlay/521e069cf402754d46060e1bafa0b18e756139df52b08a74e8b0438c61438110/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/config-reloader/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967498 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.81675161s: [/var/lib/containers/storage/overlay/bc80eb28d3232ac53dee20a3602cbc5e60a50d2d8d78d3d52d4fbe74d7d3888e/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g_62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.967517 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.634558982s: [/var/lib/containers/storage/overlay/0be7a99a9ae6a056810dde712060ca5bed577434cc98fd725791a241a8073869/diff /var/log/pods/openshift-ingress_e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv_479b82c5-9b9a-4abe-9372-38c5305bb753/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.986363 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.836954784s: [/var/lib/containers/storage/overlay/88df317889aa736ff40dff769392c7a3f3a65daa1161b79dab714b6620ef8fdc/diff /var/log/pods/openshift-ingress_e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6_667699ea-ed8e-4055-ae52-3bd87e5e966d/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.986397 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.115841935s: [/var/lib/containers/storage/overlay/1238bb420831d4a9d56b08a791557564c1a798faab9d54b5e12b037d460880d4/diff /var/log/pods/e2e-models-e2e-worker-w1-381659_premium-w1-381659-kserve-657d99b85f-9xjpn_1263f31e-87e9-4a4a-9f92-f0a03d3e0095/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.993559 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.338733341s: [/var/lib/containers/storage/overlay/d9965769b4662ff9398410690dd2b5d9c33ff7da1867f86f50d65e4211a2043f/diff /var/log/pods/openshift-ingress_e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md_7bfe9406-1e6e-4370-8b03-2b5cbb4793ae/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.997168 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.84639106s: [/var/lib/containers/storage/overlay/c4f2c3a1be8d6457b23c743b6897b2b0bf828c73d33885ad47756a57f2714bec/diff /var/log/pods/openshift-ingress_payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv_bb43505a-291d-4c54-b956-d2ae79e41263/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.997394 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.544487765s: [/var/lib/containers/storage/overlay/d62b7eedf2ace7fb78d70d23a67abe40fbf64ca5d62b0f617ed408a31d3d8753/diff /var/log/pods/cert-manager-operator_cert-manager-operator-controller-manager-7d87bcd5fd-n25hd_838c5b5f-08e0-46a8-a913-74fb85ac37c1/cert-manager-operator/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:42:43.998738 6287 container.go:588] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3633d07f_afe4_4477_bc86_d1881c6db757.slice/crio-60b05f04a4041928373c98670b3f95e4fb4da8b8ed1c1506df7edba4f28df164": error while statting cgroup v2: [unable to parse /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3633d07f_afe4_4477_bc86_d1881c6db757.slice/crio-60b05f04a4041928373c98670b3f95e4fb4da8b8ed1c1506df7edba4f28df164/memory.stat: read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3633d07f_afe4_4477_bc86_d1881c6db757.slice/crio-60b05f04a4041928373c98670b3f95e4fb4da8b8ed1c1506df7edba4f28df164/memory.stat: no such device], continuing to push stats Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.999443 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.344238268s: [/var/lib/containers/storage/overlay/685cb608c24e3a0a4940061e14abb01043393e3132563242fee391ed33b09bbe/diff ]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:43.999526 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.128853638s: [/var/lib/containers/storage/overlay/734273b17d2995a7ed56c11e1314f0a9a986412c70d884a479faff693455517e/diff /var/log/pods/e2e-models-e2e-worker-w6-b28a40_distinct-w6-b28a40-kserve-866497bdf6-xmtsl_14b2092d-b96e-485d-922d-3322cafbcf4d/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.000746 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.345514663s: [/var/lib/containers/storage/overlay/b2327c87b243e7b8cb11cddfa8928fc6157b0c59e64417be7baa7af367ebb2de/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.000976 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.345747155s: [/var/lib/containers/storage/overlay/85e5700f3e6080fbab6df2d13de1f681697328e7eb4c659ed694524962991224/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/prometheus/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.002336 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" podUID="c6c99002-ccd6-492a-b69a-198d9cb9503f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.161:50052\" within 1s: context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.008542 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.352301277s: [/var/lib/containers/storage/overlay/16f825c8f24729d0c264c63aca6539d2232c2360a37bd80fbacb33e86b721ce2/diff /var/log/pods/openshift-console_console-5fdc5c4886-h5qt9_49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26/console/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.009968 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.010008 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.353690039s: [/var/lib/containers/storage/overlay/872f3e10c3abb6805800a90c4fad92dbfc13ae7bba5f750b7f69a617123c5b96/diff ]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.021090 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:42:44.022007 6287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.325s" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.022248 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.367443098s: [/var/lib/containers/storage/overlay/80446d2a39b1a603cce33bf0a5c2b7caa6b42bc992b8f01b7c68c86192ec0701/diff /var/log/pods/openshift-ingress_e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt_980ff60d-a1fc-4f07-82ff-f916b4686bc5/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.022547 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.022570 6287 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.022615 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.036628 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.664718871s: [/var/lib/containers/storage/overlay/543d24881d833a59040092ba73e8584e483bc4f068c2cb6d6ba0be6869ab30fe/diff /var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx8f2_faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.051683 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.052700 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.054215 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.682325618s: [/var/lib/containers/storage/overlay/123fa7c6b3470fcf4950bd17d420860807a67ed94cf446ae3e3dcfb8c6e39662/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt_b00ff221-8630-4a3a-b9c3-37e90a1394fa/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.075221 6287 patch_prober.go:29] interesting pod/jobset-controller-manager-76ffc984cf-cpvzx container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.133.0.34:8081/healthz\": dial tcp 10.133.0.34:8081: connect: connection refused" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.075269 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" podUID="3633d07f-afe4-4477-bc86-d1881c6db757" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/healthz\": dial tcp 10.133.0.34:8081: connect: connection refused" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.082403 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.894496 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.909165 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.072705006s: [/var/lib/containers/storage/overlay/15c6c52814cb71a53f07b184c273f6416f7bc9937e144cbdb11d28756ab23a29/diff /var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-qscq7_e6503191-bfb0-4745-bf24-3fb040974bba/cert-manager-cainjector/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.909276 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:42:50.330374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.909300 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.254081827s: [/var/lib/containers/storage/overlay/e6b07042d45e78e4a9bb0f527d811b8f331c851da433f13e366822e38c4276be/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bjq2j_2957d06d-fec2-40f0-a4b4-b433ad6ac848/discovery/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.909335 6287 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-xqjq8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.12:9443/health\": context deadline exceeded" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.909328 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.909375 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" podUID="7059b374-3864-4296-bd7b-d77161dfca28" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.12:9443/health\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.909962 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.577084525s: [/var/lib/containers/storage/overlay/5bae94dfa336cc35c05e5cc9ca6fbe833336244a3169279e3a55e9efebe67c71/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8487c6b6d6-r86kq_c50a9903-80f6-457c-bcf9-9838a3cbdd64/manager/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.910262 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.869748133s: [/var/lib/containers/storage/overlay/dd5e499a0eabc3f8f2f78fa930c4dd95624554252bc896632342f9781b6a5a1f/diff /var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-h4rbq_1f95020b-ad58-422a-8908-3d46472f9d35/cert-manager-webhook/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.911345 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.911383 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.913742 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.914257 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.914704 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.914877 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" podUID="a7025b36-aa82-40c1-9fc1-e1e4470498c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.915363 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.915756 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.916203 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.916468 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.916952 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.917007 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.917624 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.917659 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.918189 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-87hds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.918230 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.918574 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.918610 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.919454 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.936312 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:42:44.939107 6287 logging.go:55] [core] [Channel #2831 SubChannel #2918]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.160:50052", ServerName: "10.133.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.160:50052: operation was canceled" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.929105 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" podUID="558e040b-84a9-47f0-a2b9-edcde5995e68" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.160:50052\" within 1s: context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.922024 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-prbmb" podUID="eba18759-4d3b-468a-9a93-0044457a0b93" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.919530 6287 patch_prober.go:29] interesting pod/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.939372 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.925970 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" podUID="9644b916-6603-48dc-b6e8-993bc7cec7f8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.49:50052\" within 1s: context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.919554 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.939847 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.919578 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.940524 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.949715 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" podUID="bc55462a-a012-4db1-955f-0f253626f5d2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.152:50052\" within 1s: context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.950137 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" podUID="bffc1a34-b42b-45e6-9350-31089c369391" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": dial tcp 10.133.0.26:8081: connect: connection refused" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.921550 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.950656 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.951002 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" podUID="c74aadae-e2f9-47f6-b533-88919cce7c90" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:42:44.929277 6287 logging.go:55] [core] [Channel #2829 SubChannel #2917]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.49:50052", ServerName: "10.133.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.49:50052: operation was canceled" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.951463 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" podUID="c50a9903-80f6-457c-bcf9-9838a3cbdd64" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.954061 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.51:50052\" within 1s: context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:42:44.955325 6287 logging.go:55] [core] [Channel #2833 SubChannel #2924]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.51:50052", ServerName: "10.133.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.51:50052: operation was canceled" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.958727 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" podUID="8ac2dd5a-11ff-408d-96f0-dac764508392" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.48:50052\" within 1s: context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:42:44.962356 6287 logging.go:55] [core] [Channel #2830 SubChannel #2834]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.48:50052", ServerName: "10.133.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.48:50052: operation was canceled" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:42:44.963938 6287 logging.go:55] [core] [Channel #2832 SubChannel #2923]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.152:50052", ServerName: "10.133.0.152:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.152:50052: operation was canceled" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.964148 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.964708 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:44.990445 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.017805 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": context deadline exceeded" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.017869 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.026252 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.026294 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.026335 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.026377 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.035137 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.409705669s: [/var/lib/containers/storage/overlay/1d3e1cfe75b3f7f60917e00564e2c46aa771096dfd145f7d0ac0e217d3456e8f/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn_5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.037876 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.040373 6287 trace.go:236] Trace[914853551]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" (01-Nov-2026 22:42:43.967) (total time: 1072ms): Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: Trace[914853551]: [1.072501862s] [1.072501862s] END Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.040500 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" podUID="c50a9903-80f6-457c-bcf9-9838a3cbdd64" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.048798 6287 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="console" containerStatusID={"Type":"cri-o","ID":"8104eda1e836d47fcee07a737130da6159dfcb9a01b21c13706a26b8b43dafca"} pod="openshift-console/console-5fdc5c4886-h5qt9" containerMessage="Container console failed liveness probe, will be restarted" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.052034 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.052127 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.067651 6287 trace.go:236] Trace[1744395632]: "Calculate volume metrics of multus-daemon-config for pod openshift-multus/multus-8rwzp" (01-Nov-2026 22:42:44.062) (total time: 1005ms): Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: Trace[1744395632]: [1.005576141s] [1.005576141s] END Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.073883 6287 trace.go:236] Trace[1485868646]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" (01-Nov-2026 22:42:44.012) (total time: 1061ms): Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: Trace[1485868646]: [1.061169685s] [1.061169685s] END Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.078179 6287 trace.go:236] Trace[626286328]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" (01-Nov-2026 22:42:44.021) (total time: 1056ms): Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: Trace[626286328]: [1.056334042s] [1.056334042s] END Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.086219 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.132525 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" podUID="c74aadae-e2f9-47f6-b533-88919cce7c90" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.25:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.243273 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.243341 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.472489 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" podUID="bffc1a34-b42b-45e6-9350-31089c369391" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": dial tcp 10.133.0.26:8081: connect: connection refused" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.757447 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.608119523s: [/var/lib/containers/storage/overlay/e3e720bd0912e16ba63205cdaa9a7bf88c1d28fd688798054129f9dd2a5fdae0/diff /var/log/pods/openshift-ingress_e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9_f994ce54-fa92-4360-8e3a-4e332218e556/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.764920 6287 trace.go:236] Trace[930776783]: "Calculate volume metrics of node-exporter-textfile for pod openshift-monitoring/node-exporter-8flmn" (01-Nov-2026 22:42:43.049) (total time: 2715ms): Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: Trace[930776783]: [2.715535533s] [2.715535533s] END Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:45.783467 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:47.937703 6287 trace.go:236] Trace[2109334631]: "Calculate volume metrics of metrics-client-ca for pod openshift-monitoring/node-exporter-8flmn" (01-Nov-2026 22:42:45.765) (total time: 2172ms): Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: Trace[2109334631]: [2.172589943s] [2.172589943s] END Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:47.957630 6287 trace.go:236] Trace[2132659280]: "Calculate volume metrics of cni-sysctl-allowlist for pod openshift-multus/multus-additional-cni-plugins-s86mt" (01-Nov-2026 22:42:45.687) (total time: 2269ms): Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: Trace[2132659280]: [2.269964746s] [2.269964746s] END Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:47.964817 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:47.976593 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" podUID="c74aadae-e2f9-47f6-b533-88919cce7c90" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:47.999246 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" podUID="c50a9903-80f6-457c-bcf9-9838a3cbdd64" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:48.004376 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:48.004442 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:48.048166 6287 trace.go:236] Trace[411404274]: "Calculate volume metrics of metrics-client-ca for pod openshift-monitoring/prometheus-operator-9cd89b74c-tpkmc" (01-Nov-2026 22:42:45.767) (total time: 2281ms): Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: Trace[411404274]: [2.281124338s] [2.281124338s] END Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:49.541783 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:49.541870 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:50.116247 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.626358 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.626459 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.626784 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.627066 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.627106 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.627164 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.627186 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.628168 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.162:50052: i/o timeout" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.664878 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.740214 6287 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-xqjq8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.12:9443/health\": context deadline exceeded" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.762280 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" podUID="7059b374-3864-4296-bd7b-d77161dfca28" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.12:9443/health\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.754596 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" podUID="bc55462a-a012-4db1-955f-0f253626f5d2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.152:50052\" within 1s: context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.756149 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.793078 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8487c6b6d6-r86kq" podUID="c50a9903-80f6-457c-bcf9-9838a3cbdd64" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.796763 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.092064978s: [/var/lib/containers/storage/overlay/88fe6fb7b3af97ab8b0ed27a9511fa50699982326f493a866ed9bebc544ff387/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9_bc55462a-a012-4db1-955f-0f253626f5d2/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.892588 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.896224 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.896273 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.897520 6287 patch_prober.go:29] interesting pod/jobset-controller-manager-76ffc984cf-cpvzx container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:54.173937 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:54.173996 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:54.187839 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.897556 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" podUID="3633d07f-afe4-4477-bc86-d1881c6db757" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:54.424161 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.152182 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.626431 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:53.760606 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.002695835s: [/var/lib/containers/storage/overlay/ced8608d8a7ca5d5a465c7b079364ed62b76c582e7506dbda853d2f0f21004fc/diff /var/log/pods/cert-manager_cert-manager-597b95cb84-vp7k9_2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd/cert-manager-controller/0.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.152414 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.198678 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.499166 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.533274 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.533347 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" podUID="e68fff20-1ed3-4f68-a012-444c11020805" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.533916 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:01.573335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.534397 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:08.663067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.663011 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-vp7k9" podUID="2a4d1303-cdef-4dc9-afc1-12c1af9e4dcd" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.133.0.31:9403/livez\": EOF" Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.663420 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.792069 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": context deadline exceeded" Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.795373 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.796593 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.796646 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.499250 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.824184 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" podUID="da7f0a84-0c8c-441e-8851-828d7d9949e8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.833263 6287 patch_prober.go:29] interesting pod/jobset-controller-manager-76ffc984cf-cpvzx container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.133.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.833305 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" podUID="3633d07f-afe4-4477-bc86-d1881c6db757" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.833354 6287 patch_prober.go:29] interesting pod/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:08.837236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.833373 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.838233 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" podUID="b79b3c37-2479-459b-8c93-3b00a4c9407e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.155:50052\" within 1s: context deadline exceeded" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.838480 6287 patch_prober.go:29] interesting pod/network-check-target-h5fn2 container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.133.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.838514 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-h5fn2" podUID="5029000f-c7ae-4f6f-9b0c-5240e385f697" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.133.0.4:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:08.840216 6287 logging.go:55] [core] [Channel #2948 SubChannel #2949]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.152:50052", ServerName: "10.133.0.152:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: write tcp 10.133.0.2:41452->10.133.0.152:50052: use of closed network connection Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:42:55.198207 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.845164 6287 patch_prober.go:29] interesting pod/dns-default-78mfv container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.8:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.845202 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-78mfv" podUID="8c95cb60-d629-48ef-859d-85d7893afe81" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.8:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.846713 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.649558965s: [/var/lib/containers/storage/overlay/d72d312bd8215398d67d6f5952997bc90f9c041d2e13d398b6277295996958f7/diff ]; will not log again for this container unless duration exceeds 2s Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.848073 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.848216 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.865664 6287 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9nqs9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.865700 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" podUID="f08dabd9-50ad-49a9-ba37-70a59b58c8c4" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.150:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.865951 6287 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bjq2j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.866004 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" podUID="2957d06d-fec2-40f0-a4b4-b433ad6ac848" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.102:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.867502 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" podUID="bffc1a34-b42b-45e6-9350-31089c369391" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": dial tcp 10.133.0.26:8081: connect: connection refused" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.875604 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.878057 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" podUID="bffc1a34-b42b-45e6-9350-31089c369391" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": dial tcp 10.133.0.26:8081: connect: connection refused" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.878340 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.878376 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.879533 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" podUID="f58e1b66-6f12-4a7b-93d4-02c43e499f7a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:08.879696 6287 logging.go:55] [core] [Channel #2988 SubChannel #2996]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.155:50052", ServerName: "10.133.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.155:50052: operation was canceled" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.884881 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="97258e3b-22d2-487b-932c-d8dd9432d372" containerName="prometheus" probeResult="failure" output="command timed out" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.884974 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="97258e3b-22d2-487b-932c-d8dd9432d372" containerName="prometheus" probeResult="failure" output="command timed out" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.889835 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.896078 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.901016 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.901049 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:08.907702 6287 logging.go:55] [core] [Channel #2990 SubChannel #3004]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.56:50052", ServerName: "10.133.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.56:50052: operation was canceled" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.921317 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" podUID="9644b916-6603-48dc-b6e8-993bc7cec7f8" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.927763 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" podUID="8ac2dd5a-11ff-408d-96f0-dac764508392" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.932254 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" podUID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.53:50052\" within 1s: context deadline exceeded" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.932881 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" podUID="558e040b-84a9-47f0-a2b9-edcde5995e68" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.934302 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-rx8f2" podUID="faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.934401 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" podUID="c6c99002-ccd6-492a-b69a-198d9cb9503f" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:43:08.950976 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.951653 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" podUID="b00ff221-8630-4a3a-b9c3-37e90a1394fa" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.959383 6287 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 15.219133329s: [/var/lib/containers/storage/overlay/afc629a4caea7f8e79584f286be99812d3892411218fb690aae8a40a8b707270/diff /var/log/pods/openshift-ingress_e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg_dd434389-e042-41e4-a84f-325058069d49/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.968191 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.968232 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.987331 6287 trace.go:236] Trace[1377662381]: "Calculate volume metrics of prometheus-k8s-rulefiles-1 for pod openshift-monitoring/prometheus-k8s-0" (01-Nov-2026 22:42:55.183) (total time: 13803ms): Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: Trace[1377662381]: [13.803494898s] [13.803494898s] END Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.990227 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" podUID="c46a3da2-5f01-4bba-a31a-a3adc458d938" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.992214 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" podUID="85879320-4b31-4f5e-a902-797e6f510711" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.50:50052\" within 1s: context deadline exceeded" Nov 01 22:43:09.745931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:08.992248 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:43:11.184235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:09.798545 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" podUID="c74aadae-e2f9-47f6-b533-88919cce7c90" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.25:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.184235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:09.830963 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pv9xz" podUID="c74aadae-e2f9-47f6-b533-88919cce7c90" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.184235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:09.908603 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" podUID="1f95020b-ad58-422a-8908-3d46472f9d35" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.29:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.306334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:11.306290 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.338395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:11.338368 6287 trace.go:236] Trace[1385537613]: "iptables ChainExists" (01-Nov-2026 22:43:08.923) (total time: 2414ms): Nov 01 22:43:11.338395 ip-10-0-128-55 kubenswrapper[6287]: Trace[1385537613]: [2.414485246s] [2.414485246s] END Nov 01 22:43:11.338395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:11.338369 6287 trace.go:236] Trace[2121650466]: "iptables ChainExists" (01-Nov-2026 22:43:08.821) (total time: 2516ms): Nov 01 22:43:11.338395 ip-10-0-128-55 kubenswrapper[6287]: Trace[2121650466]: [2.516955561s] [2.516955561s] END Nov 01 22:43:11.875484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:11.875266 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:11.875484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:11.875338 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.942246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:11.942214 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:11.942436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:11.942266 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.641194 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:43:12.641092 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5750b6f4_bd5e_4480_9789_153358e1335b.slice/crio-feb6c85dde5e4e179e3461916e71e7a3487a7d09ab59dc8ee269f97f083545d4\": RecentStats: unable to find data in memory cache]" Nov 01 22:43:12.803523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.803475 6287 generic.go:356] "Generic (PLEG): container finished" podID="bffc1a34-b42b-45e6-9350-31089c369391" containerID="e00dca99dfb032a88390430dfd62c2779ffa261d7fa9679687fa2c2780b37b5b" exitCode=1 Nov 01 22:43:12.902081 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:43:12.875200 6287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="28.853s" Nov 01 22:43:12.902081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.875239 6287 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:43:12.902081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.889623 6287 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:43:12.902081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.889665 6287 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:43:12.902081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.889723 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:43:12.902081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.889747 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" event={"ID":"bffc1a34-b42b-45e6-9350-31089c369391","Type":"ContainerDied","Data":"e00dca99dfb032a88390430dfd62c2779ffa261d7fa9679687fa2c2780b37b5b"} Nov 01 22:43:12.902081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.889774 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb"] Nov 01 22:43:12.902081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.899800 6287 generic.go:356] "Generic (PLEG): container finished" podID="3633d07f-afe4-4477-bc86-d1881c6db757" containerID="8cf633fd4f8871812a1d0ccb3e8615c1718422aa333a19aa882df9e98eab411c" exitCode=1 Nov 01 22:43:12.938081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.915450 6287 scope.go:122] "RemoveContainer" containerID="e00dca99dfb032a88390430dfd62c2779ffa261d7fa9679687fa2c2780b37b5b" Nov 01 22:43:12.938081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.931745 6287 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"a59572c575dcaed869ede8666ae2d549d35c6f70e161ebe869e35f7b86b6f926"} pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" containerMessage="Container manager failed liveness probe, will be restarted" Nov 01 22:43:12.938081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:12.931805 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" containerID="cri-o://a59572c575dcaed869ede8666ae2d549d35c6f70e161ebe869e35f7b86b6f926" gracePeriod=30 Nov 01 22:43:13.022135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.020637 6287 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-node-driver-registrar" containerStatusID={"Type":"cri-o","ID":"98dbc55a230ac12b17049c489e46a7a7ee745b344bd285d824b1a0acd2ff0024"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" containerMessage="Container csi-node-driver-registrar failed liveness probe, will be restarted" Nov 01 22:43:13.720755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.720717 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:13.734073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.734042 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb"] Nov 01 22:43:13.734289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.734148 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:43:13.734289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.734197 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.734692 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.734729 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" event={"ID":"3633d07f-afe4-4477-bc86-d1881c6db757","Type":"ContainerDied","Data":"8cf633fd4f8871812a1d0ccb3e8615c1718422aa333a19aa882df9e98eab411c"} Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.735154 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.735212 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.735249 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.735287 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.735335 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.735414 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.735690 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.736731 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.740057 6287 patch_prober.go:29] interesting pod/jobset-controller-manager-76ffc984cf-cpvzx container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" start-of-body= Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.740098 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" podUID="3633d07f-afe4-4477-bc86-d1881c6db757" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.745925 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.746211 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.746464 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.746676 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.749828 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.749877 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.749909 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.749938 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.749970 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750003 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750019 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750064 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750104 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vvqpg" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750401 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750421 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750457 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750491 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750512 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750553 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750601 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750632 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750662 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750694 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750709 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750739 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750768 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750798 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750816 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750866 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750913 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750952 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.750967 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.751005 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-h4rbq" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.787595 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.791260 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.797172 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.805797 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.812773 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.817729 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f2e75dd-jm5kb\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.817798 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f2e75dd-jm5kb\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.818194 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhspr\" (UniqueName: \"kubernetes.io/projected/0ad36287-197e-45f8-83f6-66780f2602c3-kube-api-access-jhspr\") pod \"maas-api-revoke-sub-6f2e75dd-jm5kb\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.837682 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.838513 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.838557 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.838708 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.840056 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.840449 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.867854 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.920575 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhspr\" (UniqueName: \"kubernetes.io/projected/0ad36287-197e-45f8-83f6-66780f2602c3-kube-api-access-jhspr\") pod \"maas-api-revoke-sub-6f2e75dd-jm5kb\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.921431 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f2e75dd-jm5kb\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.921532 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f2e75dd-jm5kb\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:13.942192 6287 scope.go:122] "RemoveContainer" containerID="8cf633fd4f8871812a1d0ccb3e8615c1718422aa333a19aa882df9e98eab411c" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:14.736734 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": context deadline exceeded" start-of-body= Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:14.736808 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": context deadline exceeded" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:14.737256 6287 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-xqjq8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.12:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:14.738452 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" podUID="7059b374-3864-4296-bd7b-d77161dfca28" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.12:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:14.776262 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:14.843301 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:14.843423 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.251401 6287 scope.go:122] "RemoveContainer" containerID="284137734d0fd6866ecc4bc82b34b4433a4ef0020bd673e1a8165a5633c0b6f2" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.257621 6287 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.279698 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f2e75dd-jm5kb\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.620153 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xqjq8" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.809234 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79"] Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.809478 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" containerID="cri-o://97a1253ce9a00c777281b5e6c318241730abc6a98f397460bcc1e8d828cef513" gracePeriod=30 Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.818002 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7"] Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.818198 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" containerID="cri-o://d1be23a457786c0a4719196b08e2054b7f729e2a3e3e4b2bd9a99a62452f18ff" gracePeriod=30 Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.820284 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.888267 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.888319 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.896387 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc"] Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.896669 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" containerID="cri-o://68951c35f590522a12708b2d25dbdfffd3cca373f196a4efb49f22a4cdad63f9" gracePeriod=30 Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.907388 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs"] Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.907596 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" containerID="cri-o://6d0eb627cee83050fae9b7bbd4f40d8d1e64245d18d737ae22fe99a7242edbc3" gracePeriod=30 Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:15.994926 6287 plugin_watcher.go:214] "Removing socket path from desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:16.012890 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx"] Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:16.013185 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" containerID="cri-o://34445fc5ff0916934f6a904bd594fef5d16b86514fd8865061e9c61958bdfe63" gracePeriod=30 Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:16.115680 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" containerID="cri-o://8104eda1e836d47fcee07a737130da6159dfcb9a01b21c13706a26b8b43dafca" gracePeriod=9 Nov 01 22:43:16.679045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:16.130458 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-node-driver-registrar" containerID="cri-o://98dbc55a230ac12b17049c489e46a7a7ee745b344bd285d824b1a0acd2ff0024" gracePeriod=27 Nov 01 22:43:19.309429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:16.742970 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f2e75dd-jm5kb\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:19.309429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:16.818651 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhspr\" (UniqueName: \"kubernetes.io/projected/0ad36287-197e-45f8-83f6-66780f2602c3-kube-api-access-jhspr\") pod \"maas-api-revoke-sub-6f2e75dd-jm5kb\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:19.309429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:16.989168 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:19.309429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:16.808303 6287 reconciler.go:150] "OperationExecutor.UnregisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"0001-01-01T00:00:00Z","UUID":"facf947f-a8e8-47fd-9ad2-ff96a584018c","Handler":{},"Name":"ebs.csi.aws.com","Endpoint":"/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"} Nov 01 22:43:19.309429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:17.381547 6287 csi_plugin.go:271] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin ebs.csi.aws.com, endpoint /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:43:19.309429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:18.042405 6287 generic.go:356] "Generic (PLEG): container finished" podID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerID="98dbc55a230ac12b17049c489e46a7a7ee745b344bd285d824b1a0acd2ff0024" exitCode=0 Nov 01 22:43:19.309429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:18.042517 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerDied","Data":"98dbc55a230ac12b17049c489e46a7a7ee745b344bd285d824b1a0acd2ff0024"} Nov 01 22:43:19.309429 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:43:18.204798 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:43:19.309429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:18.932055 6287 scope.go:122] "RemoveContainer" containerID="a8a14b8e1e4030f016cc2d2879d7bbab0be07fd5b081ca409ed7f5a8e66b1bbe" Nov 01 22:43:19.714875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:19.647879 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body= Nov 01 22:43:19.714875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:19.647919 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" Nov 01 22:43:19.879080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:19.879046 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:43:19.960161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:19.890662 6287 scope.go:122] "RemoveContainer" containerID="103696465f32c978b80c48448827f23f37a5577eda90aeecd52f7e341d31b24c" Nov 01 22:43:20.068506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.029456 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-openshift-ca\") pod \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " Nov 01 22:43:20.068506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.029823 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-xks-ca\") pod \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " Nov 01 22:43:20.068506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.029986 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31de062f-1c3b-4e81-b8e2-df7807b30cd9-kube-api-access-hq2kx\" (UniqueName: \"kubernetes.io/projected/31de062f-1c3b-4e81-b8e2-df7807b30cd9-kube-api-access-hq2kx\") pod \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\" (UID: \"31de062f-1c3b-4e81-b8e2-df7807b30cd9\") " Nov 01 22:43:20.068506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.030715 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-openshift-ca" pod "31de062f-1c3b-4e81-b8e2-df7807b30cd9" (UID: "31de062f-1c3b-4e81-b8e2-df7807b30cd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:20.068506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.031203 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:20.143578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.143024 6287 generic.go:356] "Generic (PLEG): container finished" podID="4459f450-7dab-4975-b2ac-02b042d1e004" containerID="a59572c575dcaed869ede8666ae2d549d35c6f70e161ebe869e35f7b86b6f926" exitCode=0 Nov 01 22:43:20.143578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.143103 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" event={"ID":"4459f450-7dab-4975-b2ac-02b042d1e004","Type":"ContainerDied","Data":"a59572c575dcaed869ede8666ae2d549d35c6f70e161ebe869e35f7b86b6f926"} Nov 01 22:43:20.147141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.147093 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" event={"ID":"31de062f-1c3b-4e81-b8e2-df7807b30cd9","Type":"ContainerDied","Data":"1f8dcfd3bb8bd9ca97856bde6f94153696d13774512399c7b4c34c65a523a59e"} Nov 01 22:43:20.147285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.147196 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f8dcfd3bb8bd9ca97856bde6f94153696d13774512399c7b4c34c65a523a59e" Nov 01 22:43:20.147358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.147293 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n" Nov 01 22:43:20.149954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.149559 6287 generic.go:356] "Generic (PLEG): container finished" podID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerID="8104eda1e836d47fcee07a737130da6159dfcb9a01b21c13706a26b8b43dafca" exitCode=0 Nov 01 22:43:20.149954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.149632 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdc5c4886-h5qt9" event={"ID":"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26","Type":"ContainerDied","Data":"8104eda1e836d47fcee07a737130da6159dfcb9a01b21c13706a26b8b43dafca"} Nov 01 22:43:20.231809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.231769 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-xks-ca" pod "31de062f-1c3b-4e81-b8e2-df7807b30cd9" (UID: "31de062f-1c3b-4e81-b8e2-df7807b30cd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:43:20.241438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.237318 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31de062f-1c3b-4e81-b8e2-df7807b30cd9-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:20.261553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.261496 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31de062f-1c3b-4e81-b8e2-df7807b30cd9-kube-api-access-hq2kx" pod "31de062f-1c3b-4e81-b8e2-df7807b30cd9" (UID: "31de062f-1c3b-4e81-b8e2-df7807b30cd9"). InnerVolumeSpecName "kube-api-access-hq2kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:20.302455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.302425 6287 scope.go:122] "RemoveContainer" containerID="fea52b816f6646bd672d84ac616b37982cf3ee500c87d68a8b4608788aa5f2bb" Nov 01 22:43:20.338715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.338683 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq2kx\" (UniqueName: \"kubernetes.io/projected/31de062f-1c3b-4e81-b8e2-df7807b30cd9-kube-api-access-hq2kx\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:20.864837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:20.864807 6287 scope.go:122] "RemoveContainer" containerID="46392a08783900b7a4ad6237264826b0cca0ddaa57ed33f7aa971ecd0713094c" Nov 01 22:43:21.209047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.209013 6287 scope.go:122] "RemoveContainer" containerID="796164a7463f141c90d64858814c26bc4fcd54db2f810749bb8ef748015f055d" Nov 01 22:43:21.266681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.266226 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" event={"ID":"2caef5a6-ee83-4304-b8be-583b397fb350","Type":"ContainerStarted","Data":"16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c"} Nov 01 22:43:21.266681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.266588 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:43:21.271534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.271191 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" event={"ID":"bffc1a34-b42b-45e6-9350-31089c369391","Type":"ContainerStarted","Data":"2a6f4c454336959da91527152d44b9cd9729f427bc84bd484daf7ead9df9910b"} Nov 01 22:43:21.271534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.271273 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:43:21.273184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.273092 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" event={"ID":"0f064454-385d-4336-abb8-288d642fe6fc","Type":"ContainerStarted","Data":"9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a"} Nov 01 22:43:21.273184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.273170 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:43:21.384824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.277850 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" event={"ID":"3633d07f-afe4-4477-bc86-d1881c6db757","Type":"ContainerStarted","Data":"dca3ce68e2c972532e9b073a6f956712c718e14612863547f003b6a63a583758"} Nov 01 22:43:21.384824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.278003 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:43:21.386732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.386708 6287 scope.go:122] "RemoveContainer" containerID="1a6185c9900781d0cd24eb42595533cc14e4ffc4b147beb479b25160ab5b951c" Nov 01 22:43:21.485876 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.485851 6287 scope.go:122] "RemoveContainer" containerID="0b8a427e403ff9b492122a4306bd888313cb62306dd5e5f616a5cb19d8929db0" Nov 01 22:43:21.698881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.698617 6287 scope.go:122] "RemoveContainer" containerID="0332f72a5543edee7ec77133ae9a457aec76b7949205f414f4e3b04d8a6ef647" Nov 01 22:43:21.811295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.803019 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" podStartSLOduration=26.53415534 podStartE2EDuration="1m43.291539187s" totalImagesPullingTime="1m16.757383847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:21.286080023 +0000 UTC m=+1213.934579261" watchObservedRunningTime="2026-11-01 22:43:21.291539187 +0000 UTC m=+1213.940038421" Nov 01 22:43:21.829305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:21.829241 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" podStartSLOduration=23.610725057 podStartE2EDuration="1m39.829224657s" totalImagesPullingTime="1m16.2184996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:21.82320381 +0000 UTC m=+1214.471703042" watchObservedRunningTime="2026-11-01 22:43:21.829224657 +0000 UTC m=+1214.477723894" Nov 01 22:43:22.336495 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:22.295939 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdc5c4886-h5qt9" event={"ID":"49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26","Type":"ContainerStarted","Data":"4cc9fd407be9df757ee29756a8c096706d50564794c8bdd10403852bb0fd2995"} Nov 01 22:43:24.222348 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.222295 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:43:24.235432 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.233597 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 01 22:43:24.236388 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.236362 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.247597 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.362799 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.363385 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.363439 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.371588 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.371629 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.374200 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.413239 6287 generic.go:356] "Generic (PLEG): container finished" podID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerID="68951c35f590522a12708b2d25dbdfffd3cca373f196a4efb49f22a4cdad63f9" exitCode=0 Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.413324 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" event={"ID":"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f","Type":"ContainerDied","Data":"68951c35f590522a12708b2d25dbdfffd3cca373f196a4efb49f22a4cdad63f9"} Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.415918 6287 generic.go:356] "Generic (PLEG): container finished" podID="aac045d0-c9da-4ca2-b969-804bba24373b" containerID="6d0eb627cee83050fae9b7bbd4f40d8d1e64245d18d737ae22fe99a7242edbc3" exitCode=0 Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.415968 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" event={"ID":"aac045d0-c9da-4ca2-b969-804bba24373b","Type":"ContainerDied","Data":"6d0eb627cee83050fae9b7bbd4f40d8d1e64245d18d737ae22fe99a7242edbc3"} Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.424444 6287 generic.go:356] "Generic (PLEG): container finished" podID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerID="97a1253ce9a00c777281b5e6c318241730abc6a98f397460bcc1e8d828cef513" exitCode=0 Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.424526 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" event={"ID":"ac136e0f-6211-438b-8c16-67d56e89b4e3","Type":"ContainerDied","Data":"97a1253ce9a00c777281b5e6c318241730abc6a98f397460bcc1e8d828cef513"} Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.426801 6287 generic.go:356] "Generic (PLEG): container finished" podID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerID="d1be23a457786c0a4719196b08e2054b7f729e2a3e3e4b2bd9a99a62452f18ff" exitCode=0 Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.426886 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" event={"ID":"9ee9df93-aaca-4b75-bc8b-f41d8519c256","Type":"ContainerDied","Data":"d1be23a457786c0a4719196b08e2054b7f729e2a3e3e4b2bd9a99a62452f18ff"} Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.429943 6287 generic.go:356] "Generic (PLEG): container finished" podID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerID="34445fc5ff0916934f6a904bd594fef5d16b86514fd8865061e9c61958bdfe63" exitCode=0 Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.430004 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" event={"ID":"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab","Type":"ContainerDied","Data":"34445fc5ff0916934f6a904bd594fef5d16b86514fd8865061e9c61958bdfe63"} Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.430469 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.430692 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body= Nov 01 22:43:24.481583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.430746 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" Nov 01 22:43:24.745557 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:43:24.570892 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac136e0f_6211_438b_8c16_67d56e89b4e3.slice/crio-conmon-97a1253ce9a00c777281b5e6c318241730abc6a98f397460bcc1e8d828cef513.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:43:24.745557 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:43:24.660938 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7796e1b5_a7dc_4a8f_a103_2a8aed5af17f.slice/crio-conmon-68951c35f590522a12708b2d25dbdfffd3cca373f196a4efb49f22a4cdad63f9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac136e0f_6211_438b_8c16_67d56e89b4e3.slice/crio-conmon-97a1253ce9a00c777281b5e6c318241730abc6a98f397460bcc1e8d828cef513.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11f70a0_4ce1_48ff_97fa_ec62f6e0ffab.slice/crio-conmon-34445fc5ff0916934f6a904bd594fef5d16b86514fd8865061e9c61958bdfe63.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:43:24.777649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:24.777372 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" podUID="4459f450-7dab-4975-b2ac-02b042d1e004" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/readyz\": dial tcp 10.133.0.44:8081: connect: connection refused" Nov 01 22:43:25.439561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:25.439523 6287 patch_prober.go:29] interesting pod/console-5fdc5c4886-h5qt9 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body= Nov 01 22:43:25.439972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:25.439581 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fdc5c4886-h5qt9" podUID="49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" Nov 01 22:43:25.750868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:25.750838 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:43:25.751030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:25.750886 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 22:43:25.914192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:25.912080 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 22:43:25.914192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:25.912156 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 22:43:27.252335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.223419 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 22:43:27.285014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.284641 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:27.285014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.284696 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:27.314732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.304209 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:43:27.314732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.305225 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:43:27.699838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.699806 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:43:27.722623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.722572 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-kube-api-access-64d6k\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-kube-api-access-64d6k\") pod \"ac136e0f-6211-438b-8c16-67d56e89b4e3\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " Nov 01 22:43:27.723306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.723275 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-podinfo\") pod \"ac136e0f-6211-438b-8c16-67d56e89b4e3\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " Nov 01 22:43:27.723482 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.723367 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-envoy\") pod \"ac136e0f-6211-438b-8c16-67d56e89b4e3\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " Nov 01 22:43:27.723482 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.723451 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-certs\") pod \"ac136e0f-6211-438b-8c16-67d56e89b4e3\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " Nov 01 22:43:27.725689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.723608 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-credential-socket\") pod \"ac136e0f-6211-438b-8c16-67d56e89b4e3\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " Nov 01 22:43:27.725689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.723701 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-data\") pod \"ac136e0f-6211-438b-8c16-67d56e89b4e3\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " Nov 01 22:43:27.725689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.723751 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-socket\") pod \"ac136e0f-6211-438b-8c16-67d56e89b4e3\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " Nov 01 22:43:27.725689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.723798 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac136e0f-6211-438b-8c16-67d56e89b4e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac136e0f-6211-438b-8c16-67d56e89b4e3-istiod-ca-cert\") pod \"ac136e0f-6211-438b-8c16-67d56e89b4e3\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " Nov 01 22:43:27.725689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.723922 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-token\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-token\") pod \"ac136e0f-6211-438b-8c16-67d56e89b4e3\" (UID: \"ac136e0f-6211-438b-8c16-67d56e89b4e3\") " Nov 01 22:43:27.749799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.749766 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:43:27.781589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.781549 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-socket" pod "ac136e0f-6211-438b-8c16-67d56e89b4e3" (UID: "ac136e0f-6211-438b-8c16-67d56e89b4e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:27.781786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.781579 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-credential-socket" pod "ac136e0f-6211-438b-8c16-67d56e89b4e3" (UID: "ac136e0f-6211-438b-8c16-67d56e89b4e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:27.781786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.781603 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-certs" pod "ac136e0f-6211-438b-8c16-67d56e89b4e3" (UID: "ac136e0f-6211-438b-8c16-67d56e89b4e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:27.781786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.781767 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-data" pod "ac136e0f-6211-438b-8c16-67d56e89b4e3" (UID: "ac136e0f-6211-438b-8c16-67d56e89b4e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:27.781962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.781852 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac136e0f-6211-438b-8c16-67d56e89b4e3-istiod-ca-cert" pod "ac136e0f-6211-438b-8c16-67d56e89b4e3" (UID: "ac136e0f-6211-438b-8c16-67d56e89b4e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:27.783952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.783926 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:43:27.784748 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.784723 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-envoy" pod "ac136e0f-6211-438b-8c16-67d56e89b4e3" (UID: "ac136e0f-6211-438b-8c16-67d56e89b4e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:27.823454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.784758 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-token" pod "ac136e0f-6211-438b-8c16-67d56e89b4e3" (UID: "ac136e0f-6211-438b-8c16-67d56e89b4e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:27.825824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.825792 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-certs\") pod \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " Nov 01 22:43:27.825973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.825832 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-podinfo\") pod \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " Nov 01 22:43:27.825973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.825901 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-token\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-token\") pod \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " Nov 01 22:43:27.826097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826050 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-token\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-token\") pod \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " Nov 01 22:43:27.826183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826104 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-certs\") pod \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " Nov 01 22:43:27.826304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826194 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-kube-api-access-n9k9w\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-kube-api-access-n9k9w\") pod \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " Nov 01 22:43:27.826304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826261 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istiod-ca-cert\") pod \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " Nov 01 22:43:27.826411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826343 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-podinfo\") pod \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " Nov 01 22:43:27.826411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826378 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-data\") pod \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " Nov 01 22:43:27.826515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826438 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-credential-socket\") pod \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " Nov 01 22:43:27.826515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826500 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-socket\") pod \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " Nov 01 22:43:27.826616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826539 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-credential-socket\") pod \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " Nov 01 22:43:27.826671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826612 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-envoy\") pod \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " Nov 01 22:43:27.826725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826672 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-data\") pod \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " Nov 01 22:43:27.826778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826758 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-envoy\") pod \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " Nov 01 22:43:27.826848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826813 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-kube-api-access-g6kmw\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-kube-api-access-g6kmw\") pod \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " Nov 01 22:43:27.826906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826889 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-socket\") pod \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\" (UID: \"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f\") " Nov 01 22:43:27.826963 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.826919 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istiod-ca-cert\") pod \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\" (UID: \"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab\") " Nov 01 22:43:27.827731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.827708 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:27.827836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.827733 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:27.827836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.827748 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:27.827836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.827763 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:27.827836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.827779 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac136e0f-6211-438b-8c16-67d56e89b4e3-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:27.827836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.827792 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac136e0f-6211-438b-8c16-67d56e89b4e3-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:27.827836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.827804 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:27.853074 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.842661 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-kube-api-access-64d6k" pod "ac136e0f-6211-438b-8c16-67d56e89b4e3" (UID: "ac136e0f-6211-438b-8c16-67d56e89b4e3"). InnerVolumeSpecName "kube-api-access-64d6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:27.853074 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.842719 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-podinfo" pod "ac136e0f-6211-438b-8c16-67d56e89b4e3" (UID: "ac136e0f-6211-438b-8c16-67d56e89b4e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:43:27.899475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.899441 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-socket" pod "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" (UID: "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:27.968619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.929784 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac136e0f-6211-438b-8c16-67d56e89b4e3-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:27.968619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.929810 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64d6k\" (UniqueName: \"kubernetes.io/projected/ac136e0f-6211-438b-8c16-67d56e89b4e3-kube-api-access-64d6k\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:27.968619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:27.929821 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.014546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.014517 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:43:28.030540 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.030513 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-envoy\") pod \"aac045d0-c9da-4ca2-b969-804bba24373b\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " Nov 01 22:43:28.030705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.030676 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-data\") pod \"aac045d0-c9da-4ca2-b969-804bba24373b\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " Nov 01 22:43:28.030754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.030746 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-kube-api-access-zfxm6\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-kube-api-access-zfxm6\") pod \"aac045d0-c9da-4ca2-b969-804bba24373b\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " Nov 01 22:43:28.030860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.030846 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac045d0-c9da-4ca2-b969-804bba24373b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac045d0-c9da-4ca2-b969-804bba24373b-istiod-ca-cert\") pod \"aac045d0-c9da-4ca2-b969-804bba24373b\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " Nov 01 22:43:28.030920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.030908 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-socket\") pod \"aac045d0-c9da-4ca2-b969-804bba24373b\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " Nov 01 22:43:28.031027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.031016 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-certs\") pod \"aac045d0-c9da-4ca2-b969-804bba24373b\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " Nov 01 22:43:28.031065 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.031037 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-credential-socket\") pod \"aac045d0-c9da-4ca2-b969-804bba24373b\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.031159 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aac045d0-c9da-4ca2-b969-804bba24373b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac045d0-c9da-4ca2-b969-804bba24373b-istio-podinfo\") pod \"aac045d0-c9da-4ca2-b969-804bba24373b\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.031244 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-istio-token\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-istio-token\") pod \"aac045d0-c9da-4ca2-b969-804bba24373b\" (UID: \"aac045d0-c9da-4ca2-b969-804bba24373b\") " Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.031626 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-certs" pod "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" (UID: "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.031683 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istiod-ca-cert" pod "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" (UID: "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.031701 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-data" pod "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" (UID: "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.031837 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-credential-socket" pod "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" (UID: "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.032330 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.032346 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.032356 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.032368 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.032463 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-socket" pod "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" (UID: "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.033672 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-envoy" pod "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" (UID: "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.033822 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-podinfo" pod "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" (UID: "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.033943 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-token" pod "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" (UID: "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.033978 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-kube-api-access-g6kmw" pod "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" (UID: "c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab"). InnerVolumeSpecName "kube-api-access-g6kmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.054735 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-data" pod "aac045d0-c9da-4ca2-b969-804bba24373b" (UID: "aac045d0-c9da-4ca2-b969-804bba24373b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.054993 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-socket" pod "aac045d0-c9da-4ca2-b969-804bba24373b" (UID: "aac045d0-c9da-4ca2-b969-804bba24373b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.055273 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-certs" pod "aac045d0-c9da-4ca2-b969-804bba24373b" (UID: "aac045d0-c9da-4ca2-b969-804bba24373b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.056454 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-envoy" pod "aac045d0-c9da-4ca2-b969-804bba24373b" (UID: "aac045d0-c9da-4ca2-b969-804bba24373b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.056704 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-kube-api-access-zfxm6" pod "aac045d0-c9da-4ca2-b969-804bba24373b" (UID: "aac045d0-c9da-4ca2-b969-804bba24373b"). InnerVolumeSpecName "kube-api-access-zfxm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.056840 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-istio-token" pod "aac045d0-c9da-4ca2-b969-804bba24373b" (UID: "aac045d0-c9da-4ca2-b969-804bba24373b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:28.079912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.057026 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aac045d0-c9da-4ca2-b969-804bba24373b-istio-podinfo" pod "aac045d0-c9da-4ca2-b969-804bba24373b" (UID: "aac045d0-c9da-4ca2-b969-804bba24373b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.098690 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-certs" pod "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" (UID: "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.098895 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-credential-socket" pod "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" (UID: "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.098979 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-data" pod "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" (UID: "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.099020 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istiod-ca-cert" pod "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" (UID: "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.100651 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-podinfo" pod "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" (UID: "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134247 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134267 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134277 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134286 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134294 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134303 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6kmw\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-kube-api-access-g6kmw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134312 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfxm6\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-kube-api-access-zfxm6\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134321 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134329 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134338 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134347 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134355 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac045d0-c9da-4ca2-b969-804bba24373b-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134363 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac045d0-c9da-4ca2-b969-804bba24373b-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134372 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134380 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134389 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.152958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.134397 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.153294 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-envoy" pod "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" (UID: "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.153363 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-token" pod "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" (UID: "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.153399 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac045d0-c9da-4ca2-b969-804bba24373b-istiod-ca-cert" pod "aac045d0-c9da-4ca2-b969-804bba24373b" (UID: "aac045d0-c9da-4ca2-b969-804bba24373b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.153422 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-kube-api-access-n9k9w" pod "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" (UID: "7796e1b5-a7dc-4a8f-a103-2a8aed5af17f"). InnerVolumeSpecName "kube-api-access-n9k9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.154232 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-credential-socket" pod "aac045d0-c9da-4ca2-b969-804bba24373b" (UID: "aac045d0-c9da-4ca2-b969-804bba24373b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.235648 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.235674 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac045d0-c9da-4ca2-b969-804bba24373b-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.235684 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac045d0-c9da-4ca2-b969-804bba24373b-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.235692 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.235704 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9k9w\" (UniqueName: \"kubernetes.io/projected/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f-kube-api-access-n9k9w\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.254467 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.296564 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8"] Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297402 6287 cpu_manager.go:414] "containerMap: removing container" podUID="31de062f-1c3b-4e81-b8e2-df7807b30cd9" containerName="revoke-keys" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297417 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31de062f-1c3b-4e81-b8e2-df7807b30cd9" containerName="revoke-keys" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297439 6287 cpu_manager.go:414] "containerMap: removing container" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297444 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297450 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297455 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297464 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297468 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297478 6287 cpu_manager.go:414] "containerMap: removing container" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297483 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297503 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297507 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297637 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297646 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297652 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297660 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297670 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" containerName="istio-proxy" Nov 01 22:43:28.304364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.297683 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="31de062f-1c3b-4e81-b8e2-df7807b30cd9" containerName="revoke-keys" Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.337359 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-certs\") pod \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.337421 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-token\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-token\") pod \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.337456 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-podinfo\") pod \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.337517 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-envoy\") pod \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.337564 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istiod-ca-cert\") pod \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.337597 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-socket\") pod \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.337685 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-data\") pod \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.337868 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-credential-socket\") pod \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.337968 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-kube-api-access-zjsx8\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-kube-api-access-zjsx8\") pod \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\" (UID: \"9ee9df93-aaca-4b75-bc8b-f41d8519c256\") " Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.405598 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-token" pod "9ee9df93-aaca-4b75-bc8b-f41d8519c256" (UID: "9ee9df93-aaca-4b75-bc8b-f41d8519c256"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.405659 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-podinfo" pod "9ee9df93-aaca-4b75-bc8b-f41d8519c256" (UID: "9ee9df93-aaca-4b75-bc8b-f41d8519c256"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.440896 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.448427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.440922 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.492578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.453142 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-envoy" pod "9ee9df93-aaca-4b75-bc8b-f41d8519c256" (UID: "9ee9df93-aaca-4b75-bc8b-f41d8519c256"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.492578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.453329 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv"] Nov 01 22:43:28.492578 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.453446 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.525738 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istiod-ca-cert" pod "9ee9df93-aaca-4b75-bc8b-f41d8519c256" (UID: "9ee9df93-aaca-4b75-bc8b-f41d8519c256"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.525749 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-data" pod "9ee9df93-aaca-4b75-bc8b-f41d8519c256" (UID: "9ee9df93-aaca-4b75-bc8b-f41d8519c256"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.525821 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-certs" pod "9ee9df93-aaca-4b75-bc8b-f41d8519c256" (UID: "9ee9df93-aaca-4b75-bc8b-f41d8519c256"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.525832 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-socket" pod "9ee9df93-aaca-4b75-bc8b-f41d8519c256" (UID: "9ee9df93-aaca-4b75-bc8b-f41d8519c256"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.542394 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b573f6-ec63-4be4-85bb-8db796aa13ba-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.542473 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.542635 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.542676 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.542748 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfp54\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-kube-api-access-vfp54\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.542839 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.542990 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.543018 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.543090 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.543247 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.543262 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.543270 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.543279 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.567333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.543289 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.608793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.608332 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-credential-socket" pod "9ee9df93-aaca-4b75-bc8b-f41d8519c256" (UID: "9ee9df93-aaca-4b75-bc8b-f41d8519c256"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:43:28.608793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.608427 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-kube-api-access-zjsx8" pod "9ee9df93-aaca-4b75-bc8b-f41d8519c256" (UID: "9ee9df93-aaca-4b75-bc8b-f41d8519c256"). InnerVolumeSpecName "kube-api-access-zjsx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:28.626535 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:28.611216 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad36287_197e_45f8_83f6_66780f2602c3.slice/crio-cce89f63b7aaf0e968236671875eb6da5bfb5aadc103365db026b6a965814d1a WatchSource:0}: Error finding container cce89f63b7aaf0e968236671875eb6da5bfb5aadc103365db026b6a965814d1a: Status 404 returned error can't find the container with id cce89f63b7aaf0e968236671875eb6da5bfb5aadc103365db026b6a965814d1a Nov 01 22:43:28.629877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.629847 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h"] Nov 01 22:43:28.629968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.629947 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.638844 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.638810 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" Nov 01 22:43:28.645865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.645783 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:43:28.645865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.645859 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb"] Nov 01 22:43:28.646103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.645926 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" event={"ID":"4459f450-7dab-4975-b2ac-02b042d1e004","Type":"ContainerStarted","Data":"259de5d7bbf601dd4c2009169cdd270f09b60cde7f5b808117e8e09b71960515"} Nov 01 22:43:28.646103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.645964 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc" event={"ID":"7796e1b5-a7dc-4a8f-a103-2a8aed5af17f","Type":"ContainerDied","Data":"f250cc7491e398d0cb259a542ac19b762928736247eee9227679df1d51123522"} Nov 01 22:43:28.646103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.646027 6287 scope.go:122] "RemoveContainer" containerID="68951c35f590522a12708b2d25dbdfffd3cca373f196a4efb49f22a4cdad63f9" Nov 01 22:43:28.655159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.646347 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.655159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.653405 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" Nov 01 22:43:28.660747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.657841 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs" event={"ID":"aac045d0-c9da-4ca2-b969-804bba24373b","Type":"ContainerDied","Data":"6fdd85703bfab480b373751cd442848e010078902270b036cd0bf56e55874d17"} Nov 01 22:43:28.660747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.660009 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:43:28.668441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668369 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.668532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668445 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.668532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668492 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.668660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668537 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.668660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668607 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.668660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668638 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b573f6-ec63-4be4-85bb-8db796aa13ba-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.668819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668719 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.668819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668759 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.668923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668851 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.668923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668880 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.668923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668902 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.669090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668953 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfp54\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-kube-api-access-vfp54\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.669090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.668973 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.669090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.669038 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.669090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.669083 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.672546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.669101 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.672546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.669224 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.672546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.669280 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjts7\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-kube-api-access-sjts7\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.677919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.677516 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" event={"ID":"ac136e0f-6211-438b-8c16-67d56e89b4e3","Type":"ContainerDied","Data":"52474406ee1f227e53358035f1efe4f5804bb54efaa77ed58f9e824f29eedf76"} Nov 01 22:43:28.677919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.677699 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79" Nov 01 22:43:28.682871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.682635 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-certs\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.682984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.682945 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b573f6-ec63-4be4-85bb-8db796aa13ba-istiod-ca-cert\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.689488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.683256 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.689488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.683764 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-credential-socket\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.693096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.693064 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-425870c1-openshift-default-dockercfg-lrgxq\"" Nov 01 22:43:28.705315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.705288 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee9df93-aaca-4b75-bc8b-f41d8519c256-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.705441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.705327 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjsx8\" (UniqueName: \"kubernetes.io/projected/9ee9df93-aaca-4b75-bc8b-f41d8519c256-kube-api-access-zjsx8\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:28.728907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.718319 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h"] Nov 01 22:43:28.728907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.721766 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-data\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.728907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.725345 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" Nov 01 22:43:28.728907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.726773 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7" event={"ID":"9ee9df93-aaca-4b75-bc8b-f41d8519c256","Type":"ContainerDied","Data":"c63c941962e6f1bc72ba737f3c23845237dc006f86d53c9b4b8ffaddf55d743d"} Nov 01 22:43:28.739176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.734487 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" Nov 01 22:43:28.739176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.735097 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx" event={"ID":"c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab","Type":"ContainerDied","Data":"5d68f90f739202c3e5bfa1b8cc2000f680fc27c59dbf422c009a07d7958accef"} Nov 01 22:43:28.742734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.739575 6287 scope.go:122] "RemoveContainer" containerID="6d0eb627cee83050fae9b7bbd4f40d8d1e64245d18d737ae22fe99a7242edbc3" Nov 01 22:43:28.742734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.741533 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8"] Nov 01 22:43:28.742734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.742640 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-envoy\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.745188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.745057 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-podinfo\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.749626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.749063 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfp54\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-kube-api-access-vfp54\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.749626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.749441 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-token\") pod \"e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.783267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.783026 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv"] Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809221 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809283 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809318 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809364 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-certs\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809383 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istiod-ca-cert\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809413 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809431 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-data\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809476 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjts7\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-kube-api-access-sjts7\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809568 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809603 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-envoy\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809668 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809715 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smt7l\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-kube-api-access-smt7l\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809771 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-credential-socket\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809844 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-podinfo\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809904 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.809949 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-token\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.810047 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.810089 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-socket\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.810715 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-certs\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.810982 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-data\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.811176 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-credential-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.812422 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istiod-ca-cert\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.812577 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-socket\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.814565 6287 scope.go:122] "RemoveContainer" containerID="97a1253ce9a00c777281b5e6c318241730abc6a98f397460bcc1e8d828cef513" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.815556 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-envoy\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.815975 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-podinfo\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.820564 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-token\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.824503 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjts7\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-kube-api-access-sjts7\") pod \"e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.839848 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79"] Nov 01 22:43:28.862354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.859262 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-dkq79"] Nov 01 22:43:28.899305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.870698 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7"] Nov 01 22:43:28.899305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.879488 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-xpfw7"] Nov 01 22:43:28.899305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.891251 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc"] Nov 01 22:43:28.900450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.900425 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-jq9nc"] Nov 01 22:43:28.911754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.911724 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smt7l\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-kube-api-access-smt7l\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.911881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.911778 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-credential-socket\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.911974 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.911920 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-podinfo\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.911981 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-token\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.912072 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-socket\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.912249 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-certs\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.912283 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istiod-ca-cert\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.912328 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-data\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.912396 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-credential-socket\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.912504 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-envoy\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.912575 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-socket\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.912618 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-certs\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.912770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.912738 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-data\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.928136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.928095 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs"] Nov 01 22:43:28.933143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.933104 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:28.934567 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.934542 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-ckbbs"] Nov 01 22:43:28.936711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.936668 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istiod-ca-cert\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.951944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.951911 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx"] Nov 01 22:43:28.956690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.956660 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-w45hx"] Nov 01 22:43:28.972237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.972206 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-envoy\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.972618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.972469 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-podinfo\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.972618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.972538 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-token\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.972741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.972720 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smt7l\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-kube-api-access-smt7l\") pod \"e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:28.972864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:28.972847 6287 scope.go:122] "RemoveContainer" containerID="d1be23a457786c0a4719196b08e2054b7f729e2a3e3e4b2bd9a99a62452f18ff" Nov 01 22:43:29.030581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.025765 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:29.061048 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.061017 6287 scope.go:122] "RemoveContainer" containerID="34445fc5ff0916934f6a904bd594fef5d16b86514fd8865061e9c61958bdfe63" Nov 01 22:43:29.237263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.237226 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:29.380103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.380034 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8"] Nov 01 22:43:29.469865 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:29.469833 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b573f6_ec63_4be4_85bb_8db796aa13ba.slice/crio-2bebd93aa2e18906ccf863db564227f6a94959cb0306271259d8bdc9e4b89e63 WatchSource:0}: Error finding container 2bebd93aa2e18906ccf863db564227f6a94959cb0306271259d8bdc9e4b89e63: Status 404 returned error can't find the container with id 2bebd93aa2e18906ccf863db564227f6a94959cb0306271259d8bdc9e4b89e63 Nov 01 22:43:29.480855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.480390 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:29.480855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.480486 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:29.480855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.480520 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:29.485857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.485823 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7796e1b5-a7dc-4a8f-a103-2a8aed5af17f" path="/var/lib/kubelet/pods/7796e1b5-a7dc-4a8f-a103-2a8aed5af17f/volumes" Nov 01 22:43:29.487207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.486767 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee9df93-aaca-4b75-bc8b-f41d8519c256" path="/var/lib/kubelet/pods/9ee9df93-aaca-4b75-bc8b-f41d8519c256/volumes" Nov 01 22:43:29.488550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.487668 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac045d0-c9da-4ca2-b969-804bba24373b" path="/var/lib/kubelet/pods/aac045d0-c9da-4ca2-b969-804bba24373b/volumes" Nov 01 22:43:29.488934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.488871 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac136e0f-6211-438b-8c16-67d56e89b4e3" path="/var/lib/kubelet/pods/ac136e0f-6211-438b-8c16-67d56e89b4e3/volumes" Nov 01 22:43:29.489845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.489823 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab" path="/var/lib/kubelet/pods/c11f70a0-4ce1-48ff-97fa-ec62f6e0ffab/volumes" Nov 01 22:43:29.701934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.701912 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv"] Nov 01 22:43:29.706669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.706648 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h"] Nov 01 22:43:29.755797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.755761 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" event={"ID":"0ad36287-197e-45f8-83f6-66780f2602c3","Type":"ContainerStarted","Data":"6c0c30bcd9cc14f5342ca8dab5e005c2bccd378b86e34ea21abeb8c5920be7f4"} Nov 01 22:43:29.755972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.755822 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" event={"ID":"0ad36287-197e-45f8-83f6-66780f2602c3","Type":"ContainerStarted","Data":"cce89f63b7aaf0e968236671875eb6da5bfb5aadc103365db026b6a965814d1a"} Nov 01 22:43:29.757673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.757644 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" event={"ID":"01b573f6-ec63-4be4-85bb-8db796aa13ba","Type":"ContainerStarted","Data":"1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0"} Nov 01 22:43:29.757784 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.757695 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" event={"ID":"01b573f6-ec63-4be4-85bb-8db796aa13ba","Type":"ContainerStarted","Data":"2bebd93aa2e18906ccf863db564227f6a94959cb0306271259d8bdc9e4b89e63"} Nov 01 22:43:29.766898 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:29.766871 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf0787b_8f49_43c9_99dc_1805bec7e2ac.slice/crio-c2e5fc5ed4f2d56205eb5d600bece094502e083b6336ff38c1a84a777f4983b0 WatchSource:0}: Error finding container c2e5fc5ed4f2d56205eb5d600bece094502e083b6336ff38c1a84a777f4983b0: Status 404 returned error can't find the container with id c2e5fc5ed4f2d56205eb5d600bece094502e083b6336ff38c1a84a777f4983b0 Nov 01 22:43:29.780085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.779994 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" podStartSLOduration=31.77998083 podStartE2EDuration="31.77998083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:29.776990321 +0000 UTC m=+1222.425489555" watchObservedRunningTime="2026-11-01 22:43:29.77998083 +0000 UTC m=+1222.428480064" Nov 01 22:43:29.808900 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:29.808866 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfad33b7_9986_496c_b4cd_b8b8d7a5bf83.slice/crio-55baaee079bc1624111015e8c76f2b78fdd680946fa6cff277dbefc4123dc099 WatchSource:0}: Error finding container 55baaee079bc1624111015e8c76f2b78fdd680946fa6cff277dbefc4123dc099: Status 404 returned error can't find the container with id 55baaee079bc1624111015e8c76f2b78fdd680946fa6cff277dbefc4123dc099 Nov 01 22:43:29.809047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.808929 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:29.809047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.809022 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:29.809193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.809065 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:29.812055 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.812024 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:29.825292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.812102 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:29.825292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:29.812189 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:30.775099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.775057 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerStarted","Data":"39574750647c22b8b0fa541c839dcd3551bfb3815fe3d7242f1a9261de91e5ed"} Nov 01 22:43:30.840549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.776900 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" event={"ID":"1cf0787b-8f49-43c9-99dc-1805bec7e2ac","Type":"ContainerStarted","Data":"7ebada15c1c16daddb5b0258426b36981b407ff70243bb0a733048059580eb53"} Nov 01 22:43:30.840549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.776938 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" event={"ID":"1cf0787b-8f49-43c9-99dc-1805bec7e2ac","Type":"ContainerStarted","Data":"c2e5fc5ed4f2d56205eb5d600bece094502e083b6336ff38c1a84a777f4983b0"} Nov 01 22:43:30.840549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.778648 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" event={"ID":"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83","Type":"ContainerStarted","Data":"b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9"} Nov 01 22:43:30.840549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.778694 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" event={"ID":"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83","Type":"ContainerStarted","Data":"55baaee079bc1624111015e8c76f2b78fdd680946fa6cff277dbefc4123dc099"} Nov 01 22:43:30.840549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.827522 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podStartSLOduration=97.827508383 podStartE2EDuration="1m37.827508383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:30.820900756 +0000 UTC m=+1223.469399991" watchObservedRunningTime="2026-11-01 22:43:30.827508383 +0000 UTC m=+1223.476007615" Nov 01 22:43:30.854399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.854352 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podStartSLOduration=56.854338378 podStartE2EDuration="56.854338378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:30.849194703 +0000 UTC m=+1223.497693939" watchObservedRunningTime="2026-11-01 22:43:30.854338378 +0000 UTC m=+1223.502837612" Nov 01 22:43:30.876221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.876159 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podStartSLOduration=97.876141362 podStartE2EDuration="1m37.876141362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:30.875868499 +0000 UTC m=+1223.524367736" watchObservedRunningTime="2026-11-01 22:43:30.876141362 +0000 UTC m=+1223.524640597" Nov 01 22:43:30.889243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.889208 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:30.889399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.889259 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:30.933645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.933612 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:30.934903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.934876 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:43:30.935002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:30.934921 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:43:31.027075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:31.026989 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:31.028515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:31.028477 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:43:31.028683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:31.028523 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:43:31.238308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:31.238274 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:31.239709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:31.239683 6287 patch_prober.go:29] interesting pod/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:43:31.239841 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:31.239732 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:43:31.934150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:31.934100 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:43:31.934542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:31.934173 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:43:32.027246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.027204 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:43:32.027418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.027274 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:43:32.238215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.238181 6287 patch_prober.go:29] interesting pod/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:43:32.238393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.238244 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:43:32.299252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.299222 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-z26xx" Nov 01 22:43:32.303407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.303382 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-76ffc984cf-cpvzx" Nov 01 22:43:32.366257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.366223 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj"] Nov 01 22:43:32.373234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.373204 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.401584 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.401553 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj"] Nov 01 22:43:32.517991 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.517732 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51478166-ea06-486a-b575-abf499efc600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.517991 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.517788 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms8fw\" (UniqueName: \"kubernetes.io/projected/51478166-ea06-486a-b575-abf499efc600-kube-api-access-ms8fw\") pod \"maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.518436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.518129 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51478166-ea06-486a-b575-abf499efc600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.620696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.620642 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51478166-ea06-486a-b575-abf499efc600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.621050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.621032 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51478166-ea06-486a-b575-abf499efc600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.621172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.621087 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms8fw\" (UniqueName: \"kubernetes.io/projected/51478166-ea06-486a-b575-abf499efc600-kube-api-access-ms8fw\") pod \"maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.638807 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.638740 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51478166-ea06-486a-b575-abf499efc600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.640929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.640897 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51478166-ea06-486a-b575-abf499efc600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.641231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.641205 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms8fw\" (UniqueName: \"kubernetes.io/projected/51478166-ea06-486a-b575-abf499efc600-kube-api-access-ms8fw\") pod \"maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.701587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.701544 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:32.934766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.934211 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:43:32.934766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.934287 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:43:32.941524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.940573 6287 generic.go:356] "Generic (PLEG): container finished" podID="0ad36287-197e-45f8-83f6-66780f2602c3" containerID="6c0c30bcd9cc14f5342ca8dab5e005c2bccd378b86e34ea21abeb8c5920be7f4" exitCode=0 Nov 01 22:43:32.941524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:32.941250 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" event={"ID":"0ad36287-197e-45f8-83f6-66780f2602c3","Type":"ContainerDied","Data":"6c0c30bcd9cc14f5342ca8dab5e005c2bccd378b86e34ea21abeb8c5920be7f4"} Nov 01 22:43:33.064305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:33.027454 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:43:33.064305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:33.027507 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:43:33.245644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:33.245607 6287 patch_prober.go:29] interesting pod/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:43:33.246029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:33.245998 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:43:33.755385 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:33.754194 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj"] Nov 01 22:43:33.959459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:33.959424 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:43:33.960027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:33.959996 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:43:34.025558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:34.024086 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" event={"ID":"51478166-ea06-486a-b575-abf499efc600","Type":"ContainerStarted","Data":"03580772b9d9d85cf3495b27695ccab76d09e0e6f1d7b4aa4c51a7ded1dde6c7"} Nov 01 22:43:34.052296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:34.051397 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:43:34.052296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:34.051457 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:43:34.252689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:34.252211 6287 patch_prober.go:29] interesting pod/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:43:34.252689 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:34.252274 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:43:34.935413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:34.934872 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:43:34.935413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:34.934937 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:43:34.981552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:34.953365 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:35.224197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.026791 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-openshift-ca\") pod \"0ad36287-197e-45f8-83f6-66780f2602c3\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " Nov 01 22:43:35.224197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.026871 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad36287-197e-45f8-83f6-66780f2602c3-kube-api-access-jhspr\" (UniqueName: \"kubernetes.io/projected/0ad36287-197e-45f8-83f6-66780f2602c3-kube-api-access-jhspr\") pod \"0ad36287-197e-45f8-83f6-66780f2602c3\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " Nov 01 22:43:35.224197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.026994 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-xks-ca\") pod \"0ad36287-197e-45f8-83f6-66780f2602c3\" (UID: \"0ad36287-197e-45f8-83f6-66780f2602c3\") " Nov 01 22:43:35.224197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.047505 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-openshift-ca" pod "0ad36287-197e-45f8-83f6-66780f2602c3" (UID: "0ad36287-197e-45f8-83f6-66780f2602c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:35.224197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.084511 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:43:35.224197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.084570 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:43:35.224197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.152598 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:35.269875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.262415 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-xks-ca" pod "0ad36287-197e-45f8-83f6-66780f2602c3" (UID: "0ad36287-197e-45f8-83f6-66780f2602c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:43:35.269875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.269453 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad36287-197e-45f8-83f6-66780f2602c3-kube-api-access-jhspr" pod "0ad36287-197e-45f8-83f6-66780f2602c3" (UID: "0ad36287-197e-45f8-83f6-66780f2602c3"). InnerVolumeSpecName "kube-api-access-jhspr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:35.356409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.344509 6287 patch_prober.go:29] interesting pod/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:43:35.356409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.344584 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:43:35.359711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.357620 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhspr\" (UniqueName: \"kubernetes.io/projected/0ad36287-197e-45f8-83f6-66780f2602c3-kube-api-access-jhspr\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:35.359711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.357656 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad36287-197e-45f8-83f6-66780f2602c3-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:35.935200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:35.927609 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:43:36.163831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.162350 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:43:36.163831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.162550 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:43:36.163831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.162954 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:43:36.163831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.162991 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:43:36.254050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.232361 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:36.258554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.255034 6287 patch_prober.go:29] interesting pod/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:43:36.258554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.255093 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:43:36.427960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.409789 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" Nov 01 22:43:36.490142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.489446 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb" event={"ID":"0ad36287-197e-45f8-83f6-66780f2602c3","Type":"ContainerDied","Data":"cce89f63b7aaf0e968236671875eb6da5bfb5aadc103365db026b6a965814d1a"} Nov 01 22:43:36.490142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.489519 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cce89f63b7aaf0e968236671875eb6da5bfb5aadc103365db026b6a965814d1a" Nov 01 22:43:36.490142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.489536 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" event={"ID":"51478166-ea06-486a-b575-abf499efc600","Type":"ContainerStarted","Data":"66c2c7c85b7d7c579f083896617e4cfb57dcbb29d7520a6caedea208fb6c0be2"} Nov 01 22:43:36.854220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.846205 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:36.854220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.846272 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:36.954519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.951523 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:43:36.954519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:36.951584 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:43:37.053758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:37.053714 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:43:37.054084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:37.054049 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:43:37.065880 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:43:37.060050 6287 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad36287_197e_45f8_83f6_66780f2602c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51478166_ea06_486a_b575_abf499efc600.slice/crio-66c2c7c85b7d7c579f083896617e4cfb57dcbb29d7520a6caedea208fb6c0be2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51478166_ea06_486a_b575_abf499efc600.slice/crio-conmon-66c2c7c85b7d7c579f083896617e4cfb57dcbb29d7520a6caedea208fb6c0be2.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:43:37.258292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:37.243520 6287 patch_prober.go:29] interesting pod/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:43:37.258292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:37.243577 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:43:37.662538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:37.662408 6287 generic.go:356] "Generic (PLEG): container finished" podID="51478166-ea06-486a-b575-abf499efc600" containerID="66c2c7c85b7d7c579f083896617e4cfb57dcbb29d7520a6caedea208fb6c0be2" exitCode=0 Nov 01 22:43:37.662770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:37.662689 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" event={"ID":"51478166-ea06-486a-b575-abf499efc600","Type":"ContainerDied","Data":"66c2c7c85b7d7c579f083896617e4cfb57dcbb29d7520a6caedea208fb6c0be2"} Nov 01 22:43:37.944235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:37.944147 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:43:38.100291 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:37.944424 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:43:38.107136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:38.103933 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:43:38.107136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:38.103991 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:43:38.242137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:38.238511 6287 patch_prober.go:29] interesting pod/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:43:38.242137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:38.238594 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:43:38.709182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:38.708421 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:43:38.949877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:38.938585 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:43:38.949877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:38.938653 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 22:43:39.066497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.066463 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:43:39.066753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.066727 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Nov 01 22:43:39.081149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.073624 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdc5c4886-h5qt9" Nov 01 22:43:39.149559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.149531 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:39.256890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.255247 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51478166-ea06-486a-b575-abf499efc600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51478166-ea06-486a-b575-abf499efc600-maas-api-openshift-ca\") pod \"51478166-ea06-486a-b575-abf499efc600\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " Nov 01 22:43:39.256890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.255358 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51478166-ea06-486a-b575-abf499efc600-kube-api-access-ms8fw\" (UniqueName: \"kubernetes.io/projected/51478166-ea06-486a-b575-abf499efc600-kube-api-access-ms8fw\") pod \"51478166-ea06-486a-b575-abf499efc600\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " Nov 01 22:43:39.256890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.255403 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51478166-ea06-486a-b575-abf499efc600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51478166-ea06-486a-b575-abf499efc600-maas-api-xks-ca\") pod \"51478166-ea06-486a-b575-abf499efc600\" (UID: \"51478166-ea06-486a-b575-abf499efc600\") " Nov 01 22:43:39.256890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.255871 6287 patch_prober.go:29] interesting pod/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Nov 01 22:43:39.256890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.255919 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Nov 01 22:43:39.256890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.256014 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51478166-ea06-486a-b575-abf499efc600-maas-api-openshift-ca" pod "51478166-ea06-486a-b575-abf499efc600" (UID: "51478166-ea06-486a-b575-abf499efc600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:39.256890 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.256484 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51478166-ea06-486a-b575-abf499efc600-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:39.315902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.315857 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51478166-ea06-486a-b575-abf499efc600-maas-api-xks-ca" pod "51478166-ea06-486a-b575-abf499efc600" (UID: "51478166-ea06-486a-b575-abf499efc600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:43:39.316090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.315926 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51478166-ea06-486a-b575-abf499efc600-kube-api-access-ms8fw" pod "51478166-ea06-486a-b575-abf499efc600" (UID: "51478166-ea06-486a-b575-abf499efc600"). InnerVolumeSpecName "kube-api-access-ms8fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:39.363862 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.358206 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms8fw\" (UniqueName: \"kubernetes.io/projected/51478166-ea06-486a-b575-abf499efc600-kube-api-access-ms8fw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:39.363862 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.358230 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51478166-ea06-486a-b575-abf499efc600-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:39.771747 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.771717 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6b6d674b9-llr7t" Nov 01 22:43:39.940449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.940417 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:39.940763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.940741 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:39.941472 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:39.941446 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:43:40.041587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.041498 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:40.042887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.042870 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:40.047174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.045499 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:43:40.090148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.090098 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" event={"ID":"51478166-ea06-486a-b575-abf499efc600","Type":"ContainerDied","Data":"03580772b9d9d85cf3495b27695ccab76d09e0e6f1d7b4aa4c51a7ded1dde6c7"} Nov 01 22:43:40.090375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.090174 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03580772b9d9d85cf3495b27695ccab76d09e0e6f1d7b4aa4c51a7ded1dde6c7" Nov 01 22:43:40.090375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.090123 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj" Nov 01 22:43:40.092586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.092527 6287 generic.go:356] "Generic (PLEG): container finished" podID="419eba00-1830-4c81-8c5d-912f18479177" containerID="92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af" exitCode=0 Nov 01 22:43:40.093611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.093584 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" event={"ID":"419eba00-1830-4c81-8c5d-912f18479177","Type":"ContainerDied","Data":"92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af"} Nov 01 22:43:40.094166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.094131 6287 scope.go:122] "RemoveContainer" containerID="92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af" Nov 01 22:43:40.097811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.097777 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:40.098247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.097862 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:40.098247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:40.097901 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:41.104793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:41.104765 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:41.106540 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:41.106101 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:41.106866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:41.106843 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" event={"ID":"419eba00-1830-4c81-8c5d-912f18479177","Type":"ContainerStarted","Data":"3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189"} Nov 01 22:43:41.106985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:41.106917 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:43:41.511846 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:41.511801 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:43:41.512541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:41.512520 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 01 22:43:41.512604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:41.512579 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 01 22:43:42.511084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:42.511045 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 01 22:43:42.511497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:42.511098 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 01 22:43:43.511888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:43.511851 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 01 22:43:43.512383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:43.511917 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 01 22:43:44.511611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:44.511572 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 01 22:43:44.511810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:44.511640 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 01 22:43:44.859314 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:44.857281 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:43:45.514139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:45.512392 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 01 22:43:45.514139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:45.512457 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 01 22:43:45.748186 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:43:45.748137 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:43:46.512532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:46.512494 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 01 22:43:46.513173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:46.513135 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 01 22:43:46.853187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:46.848756 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:46.853187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:46.848818 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:46.989011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:46.988976 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": dial tcp 127.0.0.1:10309: connect: connection refused" start-of-body= Nov 01 22:43:46.989322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:46.989290 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": dial tcp 127.0.0.1:10309: connect: connection refused" Nov 01 22:43:47.516241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:47.516204 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:43:47.517400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:47.517376 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:43:48.039617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.039585 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp"] Nov 01 22:43:48.046709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.046673 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6f5ade6-hlnfp"] Nov 01 22:43:48.151263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.151224 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542"] Nov 01 22:43:48.152573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.152545 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0ad36287-197e-45f8-83f6-66780f2602c3" containerName="revoke-keys" Nov 01 22:43:48.152573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.152570 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad36287-197e-45f8-83f6-66780f2602c3" containerName="revoke-keys" Nov 01 22:43:48.152739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.152581 6287 cpu_manager.go:414] "containerMap: removing container" podUID="51478166-ea06-486a-b575-abf499efc600" containerName="revoke-keys" Nov 01 22:43:48.152739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.152587 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51478166-ea06-486a-b575-abf499efc600" containerName="revoke-keys" Nov 01 22:43:48.152739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.152732 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad36287-197e-45f8-83f6-66780f2602c3" containerName="revoke-keys" Nov 01 22:43:48.152739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.152738 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="51478166-ea06-486a-b575-abf499efc600" containerName="revoke-keys" Nov 01 22:43:48.156062 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.156035 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:43:48.168440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.166689 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a9c08e\"" Nov 01 22:43:48.169766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.169741 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542"] Nov 01 22:43:48.194171 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.180783 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg"] Nov 01 22:43:48.194171 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.185792 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:43:48.194171 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.189966 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a9c08e-dockercfg-nfvph\"" Nov 01 22:43:48.201387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.201359 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg"] Nov 01 22:43:48.256799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.256744 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6d5\" (UniqueName: \"kubernetes.io/projected/3b07c554-3cc3-4163-a9fc-f0fec99f2992-kube-api-access-dq6d5\") pod \"payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542\" (UID: \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:43:48.256978 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.256907 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b07c554-3cc3-4163-a9fc-f0fec99f2992-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542\" (UID: \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:43:48.391186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.382961 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6d5\" (UniqueName: \"kubernetes.io/projected/3b07c554-3cc3-4163-a9fc-f0fec99f2992-kube-api-access-dq6d5\") pod \"payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542\" (UID: \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:43:48.391186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.383132 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b07c554-3cc3-4163-a9fc-f0fec99f2992-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542\" (UID: \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:43:48.391186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.383221 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438e322-9ace-463d-a179-391ca2864d83-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg\" (UID: \"0438e322-9ace-463d-a179-391ca2864d83\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:43:48.391186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.383632 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchr5\" (UniqueName: \"kubernetes.io/projected/0438e322-9ace-463d-a179-391ca2864d83-kube-api-access-hchr5\") pod \"payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg\" (UID: \"0438e322-9ace-463d-a179-391ca2864d83\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:43:48.391186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.385265 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b07c554-3cc3-4163-a9fc-f0fec99f2992-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542\" (UID: \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:43:48.474158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.467196 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6d5\" (UniqueName: \"kubernetes.io/projected/3b07c554-3cc3-4163-a9fc-f0fec99f2992-kube-api-access-dq6d5\") pod \"payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542\" (UID: \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:43:48.477156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.476521 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:43:48.507603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.500377 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchr5\" (UniqueName: \"kubernetes.io/projected/0438e322-9ace-463d-a179-391ca2864d83-kube-api-access-hchr5\") pod \"payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg\" (UID: \"0438e322-9ace-463d-a179-391ca2864d83\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:43:48.507603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.500700 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438e322-9ace-463d-a179-391ca2864d83-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg\" (UID: \"0438e322-9ace-463d-a179-391ca2864d83\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:43:48.538158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.523779 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438e322-9ace-463d-a179-391ca2864d83-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg\" (UID: \"0438e322-9ace-463d-a179-391ca2864d83\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:43:48.538158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.526305 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchr5\" (UniqueName: \"kubernetes.io/projected/0438e322-9ace-463d-a179-391ca2864d83-kube-api-access-hchr5\") pod \"payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg\" (UID: \"0438e322-9ace-463d-a179-391ca2864d83\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:43:48.552248 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:48.552217 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:43:49.247861 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.246515 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l"] Nov 01 22:43:49.258208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.257337 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l"] Nov 01 22:43:49.258208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.257433 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.273918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.273398 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8f2555-openshift-default-dockercfg-w7mt2\"" Nov 01 22:43:49.385375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.380472 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-certs\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.386046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.386016 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-token\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.386513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.386402 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-podinfo\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.386965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.386915 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-data\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.387233 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.387177 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-credential-socket\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.387446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.387429 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-kube-api-access-5tlpd\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.387755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.387736 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d254f-3b69-454e-a0b3-b8139db8b020-istiod-ca-cert\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.387948 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.387915 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-socket\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.388361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.388344 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-envoy\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.390530 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:49.390506 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b07c554_3cc3_4163_a9fc_f0fec99f2992.slice/crio-4fa6565de4714c1bf09999f0f0d5d2f51d9f06f87e3bd42a85df26cc2284eb4f WatchSource:0}: Error finding container 4fa6565de4714c1bf09999f0f0d5d2f51d9f06f87e3bd42a85df26cc2284eb4f: Status 404 returned error can't find the container with id 4fa6565de4714c1bf09999f0f0d5d2f51d9f06f87e3bd42a85df26cc2284eb4f Nov 01 22:43:49.444798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.444747 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542"] Nov 01 22:43:49.491477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.491437 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-data\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.491477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.491456 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b08fbbe-cd26-4b03-90c9-3d08ff469988" path="/var/lib/kubelet/pods/2b08fbbe-cd26-4b03-90c9-3d08ff469988/volumes" Nov 01 22:43:49.491731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.491494 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-credential-socket\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.491731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.491547 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-kube-api-access-5tlpd\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.491731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.491623 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d254f-3b69-454e-a0b3-b8139db8b020-istiod-ca-cert\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.491731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.491701 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-socket\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.492831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.491867 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-envoy\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.492831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.491964 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-certs\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.492831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.492047 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-token\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.492831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.492096 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-podinfo\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.493851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.493456 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d254f-3b69-454e-a0b3-b8139db8b020-istiod-ca-cert\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.493851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.493791 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-data\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.495835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.494021 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-credential-socket\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.495835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.495449 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-certs\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.495835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.495481 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-socket\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.496123 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.496081 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-podinfo\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.509341 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:49.500384 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0438e322_9ace_463d_a179_391ca2864d83.slice/crio-8a15cb92a985e57d2bb8bacdce3c3942cd4911324a0df1dbde23639ca6c1574e WatchSource:0}: Error finding container 8a15cb92a985e57d2bb8bacdce3c3942cd4911324a0df1dbde23639ca6c1574e: Status 404 returned error can't find the container with id 8a15cb92a985e57d2bb8bacdce3c3942cd4911324a0df1dbde23639ca6c1574e Nov 01 22:43:49.509341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.504404 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg"] Nov 01 22:43:49.520841 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.520789 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-token\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.522907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.522578 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-envoy\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.545449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.544910 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-kube-api-access-5tlpd\") pod \"e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.597190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.596354 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:49.605241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.601861 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58"] Nov 01 22:43:49.607873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.607848 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.643606 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.643558 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58"] Nov 01 22:43:49.694940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.694843 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:43:49.695255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.695221 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:43:49.807609 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.807554 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-l8f58\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.807797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.807710 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-l8f58\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.807797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.807775 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76d6f\" (UniqueName: \"kubernetes.io/projected/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-kube-api-access-76d6f\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-l8f58\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.910180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.910101 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-l8f58\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.910352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.910218 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-l8f58\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.911040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.910568 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76d6f\" (UniqueName: \"kubernetes.io/projected/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-kube-api-access-76d6f\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-l8f58\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.911040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.910968 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-l8f58\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.912741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.912720 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-l8f58\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.919267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.919236 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76d6f\" (UniqueName: \"kubernetes.io/projected/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-kube-api-access-76d6f\") pod \"maas-api-revoke-keys-e2e-mt-425870c1-l8f58\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.965776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.965749 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:49.975159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.975135 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l"] Nov 01 22:43:49.980244 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:49.980221 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18d254f_3b69_454e_a0b3_b8139db8b020.slice/crio-d3174db833b228a7851f1af875489c1a15ca232aef260f5b3ed133d760e127cc WatchSource:0}: Error finding container d3174db833b228a7851f1af875489c1a15ca232aef260f5b3ed133d760e127cc: Status 404 returned error can't find the container with id d3174db833b228a7851f1af875489c1a15ca232aef260f5b3ed133d760e127cc Nov 01 22:43:49.983700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.983666 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:49.983802 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.983742 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:49.983802 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:49.983780 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:43:50.212404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.212368 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" event={"ID":"0438e322-9ace-463d-a179-391ca2864d83","Type":"ContainerStarted","Data":"50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3"} Nov 01 22:43:50.212583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.212425 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" event={"ID":"0438e322-9ace-463d-a179-391ca2864d83","Type":"ContainerStarted","Data":"8a15cb92a985e57d2bb8bacdce3c3942cd4911324a0df1dbde23639ca6c1574e"} Nov 01 22:43:50.212583 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.212477 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:43:50.214645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.214606 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" event={"ID":"c18d254f-3b69-454e-a0b3-b8139db8b020","Type":"ContainerStarted","Data":"92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec"} Nov 01 22:43:50.214784 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.214672 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" event={"ID":"c18d254f-3b69-454e-a0b3-b8139db8b020","Type":"ContainerStarted","Data":"d3174db833b228a7851f1af875489c1a15ca232aef260f5b3ed133d760e127cc"} Nov 01 22:43:50.216378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.216356 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" event={"ID":"3b07c554-3cc3-4163-a9fc-f0fec99f2992","Type":"ContainerStarted","Data":"54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c"} Nov 01 22:43:50.216486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.216392 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" event={"ID":"3b07c554-3cc3-4163-a9fc-f0fec99f2992","Type":"ContainerStarted","Data":"4fa6565de4714c1bf09999f0f0d5d2f51d9f06f87e3bd42a85df26cc2284eb4f"} Nov 01 22:43:50.216486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.216411 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:43:50.239686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.239633 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" podStartSLOduration=2.239620957 podStartE2EDuration="2.239620957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:50.235608391 +0000 UTC m=+1242.884107626" watchObservedRunningTime="2026-11-01 22:43:50.239620957 +0000 UTC m=+1242.888120192" Nov 01 22:43:50.269503 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:50.269470 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13310bb1_f6f9_4bcf_a2e8_6b66e6fad162.slice/crio-6a460949af5cbd92b035d668a3906383a598b7e8dd77c616430a81e59d8548f6 WatchSource:0}: Error finding container 6a460949af5cbd92b035d668a3906383a598b7e8dd77c616430a81e59d8548f6: Status 404 returned error can't find the container with id 6a460949af5cbd92b035d668a3906383a598b7e8dd77c616430a81e59d8548f6 Nov 01 22:43:50.272476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.272400 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58"] Nov 01 22:43:50.279559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.279515 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l"] Nov 01 22:43:50.280034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.279993 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" podStartSLOduration=1.279981893 podStartE2EDuration="1.279981893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:50.267745821 +0000 UTC m=+1242.916245057" watchObservedRunningTime="2026-11-01 22:43:50.279981893 +0000 UTC m=+1242.928481172" Nov 01 22:43:50.315180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.315101 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" podStartSLOduration=2.315082945 podStartE2EDuration="2.315082945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:50.31076565 +0000 UTC m=+1242.959264886" watchObservedRunningTime="2026-11-01 22:43:50.315082945 +0000 UTC m=+1242.963582181" Nov 01 22:43:50.604187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:50.603298 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:51.230345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:51.230307 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" event={"ID":"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162","Type":"ContainerStarted","Data":"6e1da989cadace5964763a4a3e632cfee48d1b36168faa281eceadb8eb66640e"} Nov 01 22:43:51.230568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:51.230399 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" event={"ID":"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162","Type":"ContainerStarted","Data":"6a460949af5cbd92b035d668a3906383a598b7e8dd77c616430a81e59d8548f6"} Nov 01 22:43:51.249463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:51.249413 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" podStartSLOduration=2.249400436 podStartE2EDuration="2.249400436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:51.246328556 +0000 UTC m=+1243.894827788" watchObservedRunningTime="2026-11-01 22:43:51.249400436 +0000 UTC m=+1243.897899670" Nov 01 22:43:51.603826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:51.603794 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:52.242007 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:52.241972 6287 generic.go:356] "Generic (PLEG): container finished" podID="13310bb1-f6f9-4bcf-a2e8-6b66e6fad162" containerID="6e1da989cadace5964763a4a3e632cfee48d1b36168faa281eceadb8eb66640e" exitCode=0 Nov 01 22:43:52.242456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:52.242055 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" event={"ID":"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162","Type":"ContainerDied","Data":"6e1da989cadace5964763a4a3e632cfee48d1b36168faa281eceadb8eb66640e"} Nov 01 22:43:52.242571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:52.242554 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:52.243519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:52.243499 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:43:53.527823 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.527798 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:53.651316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.651236 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-kube-api-access-76d6f\" (UniqueName: \"kubernetes.io/projected/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-kube-api-access-76d6f\") pod \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " Nov 01 22:43:53.651518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.651364 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-openshift-ca\") pod \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " Nov 01 22:43:53.651518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.651400 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-xks-ca\") pod \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\" (UID: \"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162\") " Nov 01 22:43:53.651784 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.651748 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-openshift-ca" pod "13310bb1-f6f9-4bcf-a2e8-6b66e6fad162" (UID: "13310bb1-f6f9-4bcf-a2e8-6b66e6fad162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:53.652411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.652326 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:53.653425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.653400 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-kube-api-access-76d6f" pod "13310bb1-f6f9-4bcf-a2e8-6b66e6fad162" (UID: "13310bb1-f6f9-4bcf-a2e8-6b66e6fad162"). InnerVolumeSpecName "kube-api-access-76d6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:53.653539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.653503 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-xks-ca" pod "13310bb1-f6f9-4bcf-a2e8-6b66e6fad162" (UID: "13310bb1-f6f9-4bcf-a2e8-6b66e6fad162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:43:53.754671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.754636 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:53.754671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:53.754666 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76d6f\" (UniqueName: \"kubernetes.io/projected/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162-kube-api-access-76d6f\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:54.279102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.279069 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh"] Nov 01 22:43:54.279954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.279935 6287 cpu_manager.go:414] "containerMap: removing container" podUID="13310bb1-f6f9-4bcf-a2e8-6b66e6fad162" containerName="revoke-keys" Nov 01 22:43:54.279954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.279953 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13310bb1-f6f9-4bcf-a2e8-6b66e6fad162" containerName="revoke-keys" Nov 01 22:43:54.281082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.280131 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="13310bb1-f6f9-4bcf-a2e8-6b66e6fad162" containerName="revoke-keys" Nov 01 22:43:54.284557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.284542 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:43:54.289438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.289418 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8f2555\"" Nov 01 22:43:54.289524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.289503 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8f2555-dockercfg-jmd6w\"" Nov 01 22:43:54.305671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.305631 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh"] Nov 01 22:43:54.697953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.697856 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh\" (UID: \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:43:54.698629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.698597 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9k5\" (UniqueName: \"kubernetes.io/projected/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-kube-api-access-4h9k5\") pod \"payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh\" (UID: \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:43:54.861290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.858939 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh\" (UID: \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:43:54.861290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.859228 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9k5\" (UniqueName: \"kubernetes.io/projected/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-kube-api-access-4h9k5\") pod \"payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh\" (UID: \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:43:54.877259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.864617 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh\" (UID: \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:43:54.900418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.899847 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9k5\" (UniqueName: \"kubernetes.io/projected/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-kube-api-access-4h9k5\") pod \"payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh\" (UID: \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:43:54.915169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.914673 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:43:54.968188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.967187 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" event={"ID":"13310bb1-f6f9-4bcf-a2e8-6b66e6fad162","Type":"ContainerDied","Data":"6a460949af5cbd92b035d668a3906383a598b7e8dd77c616430a81e59d8548f6"} Nov 01 22:43:54.968188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.967288 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a460949af5cbd92b035d668a3906383a598b7e8dd77c616430a81e59d8548f6" Nov 01 22:43:54.968188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:54.967357 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58" Nov 01 22:43:55.279829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.279785 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" podUID="2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2" containerName="payload-pre-processing" containerID="cri-o://0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508" gracePeriod=25 Nov 01 22:43:55.298867 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:55.298830 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e3c8dc_335c_4a2b_a934_6ec7bb99d443.slice/crio-8b4bc45bf6eaa841cfc39781bd6a1c57794726ce1d469ce734b0e5660241eb46 WatchSource:0}: Error finding container 8b4bc45bf6eaa841cfc39781bd6a1c57794726ce1d469ce734b0e5660241eb46: Status 404 returned error can't find the container with id 8b4bc45bf6eaa841cfc39781bd6a1c57794726ce1d469ce734b0e5660241eb46 Nov 01 22:43:55.300385 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.300360 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh"] Nov 01 22:43:55.648310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.648282 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:43:55.791712 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.791618 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-kube-api-access-qtjgc\" (UniqueName: \"kubernetes.io/projected/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-kube-api-access-qtjgc\") pod \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\" (UID: \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\") " Nov 01 22:43:55.792192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.791813 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-plugins-config-volume\") pod \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\" (UID: \"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2\") " Nov 01 22:43:55.792446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.792423 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-plugins-config-volume" pod "2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2" (UID: "2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:55.792999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.792978 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:55.794210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.794183 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-kube-api-access-qtjgc" pod "2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2" (UID: "2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2"). InnerVolumeSpecName "kube-api-access-qtjgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:55.894619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.894588 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtjgc\" (UniqueName: \"kubernetes.io/projected/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2-kube-api-access-qtjgc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:55.980390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.980359 6287 generic.go:356] "Generic (PLEG): container finished" podID="2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2" containerID="0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508" exitCode=0 Nov 01 22:43:55.980563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.980401 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" Nov 01 22:43:55.980563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.980447 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" event={"ID":"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2","Type":"ContainerDied","Data":"0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508"} Nov 01 22:43:55.980563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.980501 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" event={"ID":"2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2","Type":"ContainerDied","Data":"52ab6f2f618bcbcce8bd2fa1bb6a5f3bdb8bb5010f5c47e203f85fbc9915d999"} Nov 01 22:43:55.980563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.980526 6287 scope.go:122] "RemoveContainer" containerID="0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508" Nov 01 22:43:55.984962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.984934 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" event={"ID":"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443","Type":"ContainerStarted","Data":"c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab"} Nov 01 22:43:55.985132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.984995 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:43:55.985132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.985018 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" event={"ID":"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443","Type":"ContainerStarted","Data":"8b4bc45bf6eaa841cfc39781bd6a1c57794726ce1d469ce734b0e5660241eb46"} Nov 01 22:43:55.997744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.997719 6287 scope.go:122] "RemoveContainer" containerID="0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508" Nov 01 22:43:55.998548 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:43:55.998512 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508\": container with ID starting with 0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508 not found: ID does not exist" containerID="0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508" Nov 01 22:43:55.998653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:55.998554 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508"} err="failed to get container status \"0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508\": rpc error: code = NotFound desc = could not find container \"0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508\": container with ID starting with 0b1d7d3e149208520e1b5aa5dee9e5389105827ce2408aca29d44721be11a508 not found: ID does not exist" Nov 01 22:43:56.003271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:56.003225 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" podStartSLOduration=2.003207946 podStartE2EDuration="2.003207946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:56.001202255 +0000 UTC m=+1248.649701494" watchObservedRunningTime="2026-11-01 22:43:56.003207946 +0000 UTC m=+1248.651707181" Nov 01 22:43:56.019524 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:56.019490 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l"] Nov 01 22:43:56.025274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:56.025250 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l"] Nov 01 22:43:56.134470 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:56.134385 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv"] Nov 01 22:43:56.161862 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:56.161828 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g"] Nov 01 22:43:56.616135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:56.615260 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee8d1a-7b89cd4c6c-8zf9l" podUID="2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.172:50052\" within 1s: context deadline exceeded" Nov 01 22:43:56.616135 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:43:56.615678 6287 logging.go:55] [core] [Channel #3381 SubChannel #3382]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.172:50052", ServerName: "10.133.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.172:50052: operation was canceled" Nov 01 22:43:56.838142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:56.833298 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=context deadline exceeded Nov 01 22:43:56.838142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:56.833361 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 01 22:43:57.020434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:57.020397 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": dial tcp 127.0.0.1:10309: connect: connection refused" start-of-body= Nov 01 22:43:57.020828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:57.020791 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": dial tcp 127.0.0.1:10309: connect: connection refused" Nov 01 22:43:57.482907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:43:57.482778 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2" path="/var/lib/kubelet/pods/2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2/volumes" Nov 01 22:44:01.135428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.135352 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" podUID="bb43505a-291d-4c54-b956-d2ae79e41263" containerName="payload-processing" containerID="cri-o://eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb" gracePeriod=25 Nov 01 22:44:01.160434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.160397 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 22:44:01.161795 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.161764 6287 generic.go:356] "Generic (PLEG): container finished" podID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerID="39574750647c22b8b0fa541c839dcd3551bfb3815fe3d7242f1a9261de91e5ed" exitCode=1 Nov 01 22:44:01.161946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.161838 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerDied","Data":"39574750647c22b8b0fa541c839dcd3551bfb3815fe3d7242f1a9261de91e5ed"} Nov 01 22:44:01.161946 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.161918 6287 scope.go:122] "RemoveContainer" containerID="98dbc55a230ac12b17049c489e46a7a7ee745b344bd285d824b1a0acd2ff0024" Nov 01 22:44:01.162217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.162186 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" podUID="62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0" containerName="payload-pre-processing" containerID="cri-o://a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e" gracePeriod=25 Nov 01 22:44:01.162569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.162532 6287 scope.go:122] "RemoveContainer" containerID="39574750647c22b8b0fa541c839dcd3551bfb3815fe3d7242f1a9261de91e5ed" Nov 01 22:44:01.234473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.234446 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:44:01.234585 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.234521 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:44:01.593378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.593351 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:44:01.602688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.602666 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:44:01.691639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.691544 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb43505a-291d-4c54-b956-d2ae79e41263-kube-api-access-cjxrg\" (UniqueName: \"kubernetes.io/projected/bb43505a-291d-4c54-b956-d2ae79e41263-kube-api-access-cjxrg\") pod \"bb43505a-291d-4c54-b956-d2ae79e41263\" (UID: \"bb43505a-291d-4c54-b956-d2ae79e41263\") " Nov 01 22:44:01.691639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.691612 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-plugins-config-volume\") pod \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\" (UID: \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\") " Nov 01 22:44:01.691878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.691727 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-kube-api-access-24fp4\" (UniqueName: \"kubernetes.io/projected/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-kube-api-access-24fp4\") pod \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\" (UID: \"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0\") " Nov 01 22:44:01.691878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.691810 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb43505a-291d-4c54-b956-d2ae79e41263-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43505a-291d-4c54-b956-d2ae79e41263-plugins-config-volume\") pod \"bb43505a-291d-4c54-b956-d2ae79e41263\" (UID: \"bb43505a-291d-4c54-b956-d2ae79e41263\") " Nov 01 22:44:01.692096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.692064 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-plugins-config-volume" pod "62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0" (UID: "62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:01.692459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.692426 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb43505a-291d-4c54-b956-d2ae79e41263-plugins-config-volume" pod "bb43505a-291d-4c54-b956-d2ae79e41263" (UID: "bb43505a-291d-4c54-b956-d2ae79e41263"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:01.692955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.692932 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:01.693029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.692962 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb43505a-291d-4c54-b956-d2ae79e41263-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:01.697421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.697391 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-kube-api-access-24fp4" pod "62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0" (UID: "62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0"). InnerVolumeSpecName "kube-api-access-24fp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:01.697610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.697587 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb43505a-291d-4c54-b956-d2ae79e41263-kube-api-access-cjxrg" pod "bb43505a-291d-4c54-b956-d2ae79e41263" (UID: "bb43505a-291d-4c54-b956-d2ae79e41263"). InnerVolumeSpecName "kube-api-access-cjxrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:01.794964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.794928 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjxrg\" (UniqueName: \"kubernetes.io/projected/bb43505a-291d-4c54-b956-d2ae79e41263-kube-api-access-cjxrg\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:01.794964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:01.794963 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fp4\" (UniqueName: \"kubernetes.io/projected/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0-kube-api-access-24fp4\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:02.181735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.181702 6287 generic.go:356] "Generic (PLEG): container finished" podID="bb43505a-291d-4c54-b956-d2ae79e41263" containerID="eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb" exitCode=0 Nov 01 22:44:02.182285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.181783 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" event={"ID":"bb43505a-291d-4c54-b956-d2ae79e41263","Type":"ContainerDied","Data":"eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb"} Nov 01 22:44:02.182285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.181793 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" Nov 01 22:44:02.182285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.181900 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv" event={"ID":"bb43505a-291d-4c54-b956-d2ae79e41263","Type":"ContainerDied","Data":"1be3376d6bd9ba229886cde92e08004b217b23b29812db76cbc183c3163182fe"} Nov 01 22:44:02.182285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.182033 6287 scope.go:122] "RemoveContainer" containerID="eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb" Nov 01 22:44:02.184859 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.184839 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 22:44:02.185647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.185620 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerStarted","Data":"ec1748ce89358a17f35e44f8058bbc96109d098cf153a284b4614f91c46b382e"} Nov 01 22:44:02.187319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.187300 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" Nov 01 22:44:02.187425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.187329 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" event={"ID":"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0","Type":"ContainerDied","Data":"a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e"} Nov 01 22:44:02.187425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.187300 6287 generic.go:356] "Generic (PLEG): container finished" podID="62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0" containerID="a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e" exitCode=0 Nov 01 22:44:02.187531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.187456 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g" event={"ID":"62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0","Type":"ContainerDied","Data":"3ad1e5202c67fed4ce8f060707a5d0e93d6929fb59b16e9a4a5c3df26ded5d19"} Nov 01 22:44:02.196374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.196355 6287 scope.go:122] "RemoveContainer" containerID="eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb" Nov 01 22:44:02.196694 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:02.196664 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb\": container with ID starting with eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb not found: ID does not exist" containerID="eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb" Nov 01 22:44:02.196902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.196702 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb"} err="failed to get container status \"eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb\": rpc error: code = NotFound desc = could not find container \"eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb\": container with ID starting with eb281cc2b660537230f8cb62affa55092f7005734f6a88b867c2aa91ae1105bb not found: ID does not exist" Nov 01 22:44:02.196902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.196727 6287 scope.go:122] "RemoveContainer" containerID="a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e" Nov 01 22:44:02.212353 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.212336 6287 scope.go:122] "RemoveContainer" containerID="a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e" Nov 01 22:44:02.212616 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:02.212594 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e\": container with ID starting with a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e not found: ID does not exist" containerID="a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e" Nov 01 22:44:02.212663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.212622 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e"} err="failed to get container status \"a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e\": rpc error: code = NotFound desc = could not find container \"a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e\": container with ID starting with a0f4bf9f64fe48f63ee59d637111a1e040f1f15aeab2d52f4edb25213130b08e not found: ID does not exist" Nov 01 22:44:02.226520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.226489 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv"] Nov 01 22:44:02.231601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.231580 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1d4be-86f687689c-xkndv"] Nov 01 22:44:02.240448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.240424 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g"] Nov 01 22:44:02.245158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:02.245138 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1d4be-64bd44bd75-8r48g"] Nov 01 22:44:03.474472 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:03.474437 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0" path="/var/lib/kubelet/pods/62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0/volumes" Nov 01 22:44:03.475044 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:03.475028 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb43505a-291d-4c54-b956-d2ae79e41263" path="/var/lib/kubelet/pods/bb43505a-291d-4c54-b956-d2ae79e41263/volumes" Nov 01 22:44:05.394401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:05.394367 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n"] Nov 01 22:44:05.402720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:05.400945 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a1d4be-mg58n"] Nov 01 22:44:05.473490 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:05.473456 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31de062f-1c3b-4e81-b8e2-df7807b30cd9" path="/var/lib/kubelet/pods/31de062f-1c3b-4e81-b8e2-df7807b30cd9/volumes" Nov 01 22:44:06.832510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:06.831335 6287 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9kxlv container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=context deadline exceeded Nov 01 22:44:06.832510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:06.831414 6287 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" probeResult="failure" output="HTTP probe failed with statuscode: 500" Nov 01 22:44:06.832510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:06.831467 6287 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" Nov 01 22:44:06.833446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:06.833406 6287 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-driver" containerStatusID={"Type":"cri-o","ID":"5a296f4606948844815c75184cb85207b6e661d2c918032f53b2709817520594"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" containerMessage="Container csi-driver failed liveness probe, will be restarted" Nov 01 22:44:06.833597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:06.833550 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" podUID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerName="csi-driver" containerID="cri-o://5a296f4606948844815c75184cb85207b6e661d2c918032f53b2709817520594" gracePeriod=30 Nov 01 22:44:07.027932 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.027909 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:44:07.312692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.312669 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 22:44:07.336147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.336088 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 22:44:07.336292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.336179 6287 generic.go:356] "Generic (PLEG): container finished" podID="0e214ead-da27-47f8-8196-34d885b7b4dc" containerID="5a296f4606948844815c75184cb85207b6e661d2c918032f53b2709817520594" exitCode=2 Nov 01 22:44:07.336292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.336251 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerDied","Data":"5a296f4606948844815c75184cb85207b6e661d2c918032f53b2709817520594"} Nov 01 22:44:07.336391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.336311 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9kxlv" event={"ID":"0e214ead-da27-47f8-8196-34d885b7b4dc","Type":"ContainerStarted","Data":"606b68d0243675062d7d564335eda4fe7f7f9df1deefd022b5fe358fac213c8c"} Nov 01 22:44:07.615933 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.615892 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt"] Nov 01 22:44:07.618046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.618014 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2" containerName="payload-pre-processing" Nov 01 22:44:07.618228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.618212 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2" containerName="payload-pre-processing" Nov 01 22:44:07.618322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.618312 6287 cpu_manager.go:414] "containerMap: removing container" podUID="bb43505a-291d-4c54-b956-d2ae79e41263" containerName="payload-processing" Nov 01 22:44:07.618412 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.618402 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb43505a-291d-4c54-b956-d2ae79e41263" containerName="payload-processing" Nov 01 22:44:07.618499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.618490 6287 cpu_manager.go:414] "containerMap: removing container" podUID="62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0" containerName="payload-pre-processing" Nov 01 22:44:07.618576 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.618554 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0" containerName="payload-pre-processing" Nov 01 22:44:07.618979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.618957 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4f8ba9-bc89-47f5-bdc2-95d91ef94af2" containerName="payload-pre-processing" Nov 01 22:44:07.619020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.618982 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb43505a-291d-4c54-b956-d2ae79e41263" containerName="payload-processing" Nov 01 22:44:07.619020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.618989 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c11c9a-12c9-4ecc-9466-bf1b8b1d32d0" containerName="payload-pre-processing" Nov 01 22:44:07.627894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.627857 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:07.649707 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.649670 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt"] Nov 01 22:44:07.651315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.651291 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:07.651315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.651323 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:44:07.760659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.760629 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7wgj\" (UniqueName: \"kubernetes.io/projected/3f222087-18d6-46b2-9d37-0c4c510d8589-kube-api-access-h7wgj\") pod \"maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:07.760826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.760765 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:07.760826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.760795 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:07.872214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.868992 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:07.872214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.869026 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:07.872214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.869341 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7wgj\" (UniqueName: \"kubernetes.io/projected/3f222087-18d6-46b2-9d37-0c4c510d8589-kube-api-access-h7wgj\") pod \"maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:07.872214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.870593 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:07.953571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.953503 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:07.953751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:07.953717 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7wgj\" (UniqueName: \"kubernetes.io/projected/3f222087-18d6-46b2-9d37-0c4c510d8589-kube-api-access-h7wgj\") pod \"maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:08.245454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:08.245424 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:08.597297 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:44:08.597265 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f222087_18d6_46b2_9d37_0c4c510d8589.slice/crio-858c28b4e58b3015bff350b719ae84ab135a5b583fb96be2954c9f306b851d00 WatchSource:0}: Error finding container 858c28b4e58b3015bff350b719ae84ab135a5b583fb96be2954c9f306b851d00: Status 404 returned error can't find the container with id 858c28b4e58b3015bff350b719ae84ab135a5b583fb96be2954c9f306b851d00 Nov 01 22:44:08.600845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:08.600818 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt"] Nov 01 22:44:08.954452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:08.953286 6287 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:44:09.360185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:09.360142 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" event={"ID":"3f222087-18d6-46b2-9d37-0c4c510d8589","Type":"ContainerStarted","Data":"84f9d005f085a3286414dbf62d982cc72505617fd8882c522b5a9bd38594f431"} Nov 01 22:44:09.360378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:09.360205 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" event={"ID":"3f222087-18d6-46b2-9d37-0c4c510d8589","Type":"ContainerStarted","Data":"858c28b4e58b3015bff350b719ae84ab135a5b583fb96be2954c9f306b851d00"} Nov 01 22:44:09.381634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:09.381577 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" podStartSLOduration=2.381561343 podStartE2EDuration="2.381561343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:09.378569465 +0000 UTC m=+1262.027068701" watchObservedRunningTime="2026-11-01 22:44:09.381561343 +0000 UTC m=+1262.030060577" Nov 01 22:44:09.721352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:09.721270 6287 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:44:08.954405802Z","UUID":"5016dd71-f207-4822-8579-cee88e7d8d2a","Handler":null,"Name":"","Endpoint":""} Nov 01 22:44:09.722368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:09.722346 6287 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:44:09.722484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:09.722375 6287 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:44:10.503136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:10.502145 6287 generic.go:356] "Generic (PLEG): container finished" podID="3f222087-18d6-46b2-9d37-0c4c510d8589" containerID="84f9d005f085a3286414dbf62d982cc72505617fd8882c522b5a9bd38594f431" exitCode=0 Nov 01 22:44:10.503136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:10.502288 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" event={"ID":"3f222087-18d6-46b2-9d37-0c4c510d8589","Type":"ContainerDied","Data":"84f9d005f085a3286414dbf62d982cc72505617fd8882c522b5a9bd38594f431"} Nov 01 22:44:10.863860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:10.863769 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj"] Nov 01 22:44:10.875277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:10.875240 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee8d1a-2grrj"] Nov 01 22:44:11.474215 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.474179 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51478166-ea06-486a-b575-abf499efc600" path="/var/lib/kubelet/pods/51478166-ea06-486a-b575-abf499efc600/volumes" Nov 01 22:44:11.694942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.694920 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:11.815344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.815310 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f222087-18d6-46b2-9d37-0c4c510d8589-kube-api-access-h7wgj\" (UniqueName: \"kubernetes.io/projected/3f222087-18d6-46b2-9d37-0c4c510d8589-kube-api-access-h7wgj\") pod \"3f222087-18d6-46b2-9d37-0c4c510d8589\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " Nov 01 22:44:11.815549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.815443 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-openshift-ca\") pod \"3f222087-18d6-46b2-9d37-0c4c510d8589\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " Nov 01 22:44:11.815652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.815626 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-xks-ca\") pod \"3f222087-18d6-46b2-9d37-0c4c510d8589\" (UID: \"3f222087-18d6-46b2-9d37-0c4c510d8589\") " Nov 01 22:44:11.816106 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.816071 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-openshift-ca" pod "3f222087-18d6-46b2-9d37-0c4c510d8589" (UID: "3f222087-18d6-46b2-9d37-0c4c510d8589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:11.816538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.816517 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:11.819170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.819093 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f222087-18d6-46b2-9d37-0c4c510d8589-kube-api-access-h7wgj" pod "3f222087-18d6-46b2-9d37-0c4c510d8589" (UID: "3f222087-18d6-46b2-9d37-0c4c510d8589"). InnerVolumeSpecName "kube-api-access-h7wgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:11.819463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.819440 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-xks-ca" pod "3f222087-18d6-46b2-9d37-0c4c510d8589" (UID: "3f222087-18d6-46b2-9d37-0c4c510d8589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:11.918621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.918589 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7wgj\" (UniqueName: \"kubernetes.io/projected/3f222087-18d6-46b2-9d37-0c4c510d8589-kube-api-access-h7wgj\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:11.918621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:11.918620 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f222087-18d6-46b2-9d37-0c4c510d8589-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:12.397206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:12.397165 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp"] Nov 01 22:44:12.463138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:12.462315 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h"] Nov 01 22:44:12.532934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:12.532889 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx"] Nov 01 22:44:12.554389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:12.554342 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" event={"ID":"3f222087-18d6-46b2-9d37-0c4c510d8589","Type":"ContainerDied","Data":"858c28b4e58b3015bff350b719ae84ab135a5b583fb96be2954c9f306b851d00"} Nov 01 22:44:12.554685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:12.554416 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858c28b4e58b3015bff350b719ae84ab135a5b583fb96be2954c9f306b851d00" Nov 01 22:44:12.554685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:12.554432 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt" Nov 01 22:44:15.639498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.639298 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x"] Nov 01 22:44:15.640507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.640202 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" containerID="cri-o://1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f" gracePeriod=30 Nov 01 22:44:15.644485 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:15.644450 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:44:15.649048 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.649019 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94"] Nov 01 22:44:15.660101 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.659449 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg"] Nov 01 22:44:15.660629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.660336 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" containerID="cri-o://53d5a8711670c52bcb680f0f13dc3f936f8d7414d17a1766054f0c9bc35f3c52" gracePeriod=30 Nov 01 22:44:15.660629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.660545 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" containerID="cri-o://b665e428280f3684fcf032d352586a1f2126a289089417df448d1c6d1b555a5f" gracePeriod=30 Nov 01 22:44:15.667382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.667351 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9"] Nov 01 22:44:15.667938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.667862 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" containerID="cri-o://a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c" gracePeriod=30 Nov 01 22:44:15.675327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.675301 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4"] Nov 01 22:44:15.675509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.675486 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" containerID="cri-o://3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e" gracePeriod=30 Nov 01 22:44:15.681197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.681169 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs"] Nov 01 22:44:15.681403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.681381 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" containerID="cri-o://9f7e2056f648d663d49db9b855050a3c04096b5fb56fe55daccd38f223f6caff" gracePeriod=30 Nov 01 22:44:15.907227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.905947 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:44:15.907227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:15.906007 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:44:16.134554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.134518 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz"] Nov 01 22:44:16.135604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.135578 6287 cpu_manager.go:414] "containerMap: removing container" podUID="3f222087-18d6-46b2-9d37-0c4c510d8589" containerName="revoke-keys" Nov 01 22:44:16.135604 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.135601 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f222087-18d6-46b2-9d37-0c4c510d8589" containerName="revoke-keys" Nov 01 22:44:16.135888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.135873 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f222087-18d6-46b2-9d37-0c4c510d8589" containerName="revoke-keys" Nov 01 22:44:16.139094 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.139072 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.149493 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.149434 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:16.150616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.149517 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:44:16.155865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.155840 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz"] Nov 01 22:44:16.222635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.222549 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kbs\" (UniqueName: \"kubernetes.io/projected/58565c64-04fa-4507-b9c7-058493f53a38-kube-api-access-b9kbs\") pod \"maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.222822 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.222800 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.223152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.223099 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.325587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.325445 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.325796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.325652 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kbs\" (UniqueName: \"kubernetes.io/projected/58565c64-04fa-4507-b9c7-058493f53a38-kube-api-access-b9kbs\") pod \"maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.326500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.325895 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.326885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.326859 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.328838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.328818 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.339221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.339193 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kbs\" (UniqueName: \"kubernetes.io/projected/58565c64-04fa-4507-b9c7-058493f53a38-kube-api-access-b9kbs\") pod \"maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.459279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.459248 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:16.756168 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:44:16.756139 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58565c64_04fa_4507_b9c7_058493f53a38.slice/crio-5e4b29e6ad89aa379c0aa698076ce992adf2b76a3cdec96ad47bfbe4a1b22580 WatchSource:0}: Error finding container 5e4b29e6ad89aa379c0aa698076ce992adf2b76a3cdec96ad47bfbe4a1b22580: Status 404 returned error can't find the container with id 5e4b29e6ad89aa379c0aa698076ce992adf2b76a3cdec96ad47bfbe4a1b22580 Nov 01 22:44:16.756740 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:16.756714 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz"] Nov 01 22:44:17.662545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.662501 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" event={"ID":"58565c64-04fa-4507-b9c7-058493f53a38","Type":"ContainerStarted","Data":"81549149daa3f24be3a5bce597dd54f439f62555d510d8a18332d4559ee065a8"} Nov 01 22:44:17.662766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.662569 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" event={"ID":"58565c64-04fa-4507-b9c7-058493f53a38","Type":"ContainerStarted","Data":"5e4b29e6ad89aa379c0aa698076ce992adf2b76a3cdec96ad47bfbe4a1b22580"} Nov 01 22:44:17.693539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.693471 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" podStartSLOduration=1.693450758 podStartE2EDuration="1.693450758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:17.68941869 +0000 UTC m=+1270.337917926" watchObservedRunningTime="2026-11-01 22:44:17.693450758 +0000 UTC m=+1270.341949995" Nov 01 22:44:17.735249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.735205 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm"] Nov 01 22:44:17.740126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.740082 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:17.753427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.750738 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ae0e3a-dockercfg-sl6vn\"" Nov 01 22:44:17.753427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.752477 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjl8\" (UniqueName: \"kubernetes.io/projected/7a56c85c-8295-422a-ba5b-6744fa8fae61-kube-api-access-bkjl8\") pod \"payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm\" (UID: \"7a56c85c-8295-422a-ba5b-6744fa8fae61\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:17.753427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.752837 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56c85c-8295-422a-ba5b-6744fa8fae61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm\" (UID: \"7a56c85c-8295-422a-ba5b-6744fa8fae61\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:17.759480 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.759366 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm"] Nov 01 22:44:17.789414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.789367 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ae0e3a\"" Nov 01 22:44:17.882137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.855429 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56c85c-8295-422a-ba5b-6744fa8fae61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm\" (UID: \"7a56c85c-8295-422a-ba5b-6744fa8fae61\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:17.882137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.855618 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjl8\" (UniqueName: \"kubernetes.io/projected/7a56c85c-8295-422a-ba5b-6744fa8fae61-kube-api-access-bkjl8\") pod \"payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm\" (UID: \"7a56c85c-8295-422a-ba5b-6744fa8fae61\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:17.882137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:17.857748 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56c85c-8295-422a-ba5b-6744fa8fae61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm\" (UID: \"7a56c85c-8295-422a-ba5b-6744fa8fae61\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:18.028437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.027703 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjl8\" (UniqueName: \"kubernetes.io/projected/7a56c85c-8295-422a-ba5b-6744fa8fae61-kube-api-access-bkjl8\") pod \"payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm\" (UID: \"7a56c85c-8295-422a-ba5b-6744fa8fae61\") " pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:18.077703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.077632 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:18.367329 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.367246 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf"] Nov 01 22:44:18.390764 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.390732 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj"] Nov 01 22:44:18.438673 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:44:18.438639 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a56c85c_8295_422a_ba5b_6744fa8fae61.slice/crio-042ea8f4831f3fb4e3867f53cb926930f8ab233b357f367942c502613557147f WatchSource:0}: Error finding container 042ea8f4831f3fb4e3867f53cb926930f8ab233b357f367942c502613557147f: Status 404 returned error can't find the container with id 042ea8f4831f3fb4e3867f53cb926930f8ab233b357f367942c502613557147f Nov 01 22:44:18.439810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.439781 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm"] Nov 01 22:44:18.562744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.562712 6287 patch_prober.go:29] interesting pod/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:44:18.562886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.562772 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:44:18.886086 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.886046 6287 generic.go:356] "Generic (PLEG): container finished" podID="58565c64-04fa-4507-b9c7-058493f53a38" containerID="81549149daa3f24be3a5bce597dd54f439f62555d510d8a18332d4559ee065a8" exitCode=0 Nov 01 22:44:18.886947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.886908 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" event={"ID":"58565c64-04fa-4507-b9c7-058493f53a38","Type":"ContainerDied","Data":"81549149daa3f24be3a5bce597dd54f439f62555d510d8a18332d4559ee065a8"} Nov 01 22:44:18.909252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.908950 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" event={"ID":"7a56c85c-8295-422a-ba5b-6744fa8fae61","Type":"ContainerStarted","Data":"4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f"} Nov 01 22:44:18.909252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.909025 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:18.909252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:18.909056 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" event={"ID":"7a56c85c-8295-422a-ba5b-6744fa8fae61","Type":"ContainerStarted","Data":"042ea8f4831f3fb4e3867f53cb926930f8ab233b357f367942c502613557147f"} Nov 01 22:44:19.023160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:19.022969 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" podStartSLOduration=2.022949238 podStartE2EDuration="2.022949238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:19.007395805 +0000 UTC m=+1271.655895042" watchObservedRunningTime="2026-11-01 22:44:19.022949238 +0000 UTC m=+1271.671448476" Nov 01 22:44:20.381814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:20.381783 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8"] Nov 01 22:44:20.382267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:20.382024 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" containerID="cri-o://1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0" gracePeriod=30 Nov 01 22:44:20.391391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:20.391359 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2"] Nov 01 22:44:20.391627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:20.391601 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" containerID="cri-o://3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189" gracePeriod=30 Nov 01 22:44:20.905624 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:20.905586 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj"] Nov 01 22:44:20.935168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:20.931349 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj"] Nov 01 22:44:20.935168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:20.931473 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.040412 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.040314 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.040730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.040702 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28r9p\" (UniqueName: \"kubernetes.io/projected/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-kube-api-access-28r9p\") pod \"maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.044736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.044023 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.152744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.149813 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.152744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.149948 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28r9p\" (UniqueName: \"kubernetes.io/projected/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-kube-api-access-28r9p\") pod \"maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.154785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.153341 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.154785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.154253 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.157227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.156084 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:21.169436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.169408 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28r9p\" (UniqueName: \"kubernetes.io/projected/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-kube-api-access-28r9p\") pod \"maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.177868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.177815 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.214588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.213769 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:44:21.227299 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.227209 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:44:21.257255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.256690 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-socket\") pod \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " Nov 01 22:44:21.257635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.256981 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-socket" pod "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" (UID: "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.257792 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.257775 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-data\") pod \"f994ce54-fa92-4360-8e3a-4e332218e556\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " Nov 01 22:44:21.258297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.258082 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-data" pod "f994ce54-fa92-4360-8e3a-4e332218e556" (UID: "f994ce54-fa92-4360-8e3a-4e332218e556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.258417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.258402 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-openshift-ca\") pod \"58565c64-04fa-4507-b9c7-058493f53a38\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " Nov 01 22:44:21.258909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.258722 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-openshift-ca" pod "58565c64-04fa-4507-b9c7-058493f53a38" (UID: "58565c64-04fa-4507-b9c7-058493f53a38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:21.259383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.258866 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58565c64-04fa-4507-b9c7-058493f53a38-kube-api-access-b9kbs\" (UniqueName: \"kubernetes.io/projected/58565c64-04fa-4507-b9c7-058493f53a38-kube-api-access-b9kbs\") pod \"58565c64-04fa-4507-b9c7-058493f53a38\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " Nov 01 22:44:21.259611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.259595 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-istio-token\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-istio-token\") pod \"f994ce54-fa92-4360-8e3a-4e332218e556\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " Nov 01 22:44:21.260201 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.260174 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-certs\") pod \"f994ce54-fa92-4360-8e3a-4e332218e556\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " Nov 01 22:44:21.260707 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.260503 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-certs" pod "f994ce54-fa92-4360-8e3a-4e332218e556" (UID: "f994ce54-fa92-4360-8e3a-4e332218e556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.261184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.261164 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-kube-api-access-5rb2x\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-kube-api-access-5rb2x\") pod \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " Nov 01 22:44:21.261651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.261635 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-envoy\") pod \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " Nov 01 22:44:21.262065 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.262049 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-certs\") pod \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " Nov 01 22:44:21.262523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.262506 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f994ce54-fa92-4360-8e3a-4e332218e556-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f994ce54-fa92-4360-8e3a-4e332218e556-istiod-ca-cert\") pod \"f994ce54-fa92-4360-8e3a-4e332218e556\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " Nov 01 22:44:21.262653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.262639 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f994ce54-fa92-4360-8e3a-4e332218e556-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f994ce54-fa92-4360-8e3a-4e332218e556-istio-podinfo\") pod \"f994ce54-fa92-4360-8e3a-4e332218e556\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " Nov 01 22:44:21.263135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.263101 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-token\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-token\") pod \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " Nov 01 22:44:21.263701 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.261559 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58565c64-04fa-4507-b9c7-058493f53a38-kube-api-access-b9kbs" pod "58565c64-04fa-4507-b9c7-058493f53a38" (UID: "58565c64-04fa-4507-b9c7-058493f53a38"). InnerVolumeSpecName "kube-api-access-b9kbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.263807 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.262420 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-certs" pod "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" (UID: "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.263883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.262568 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-istio-token" pod "f994ce54-fa92-4360-8e3a-4e332218e556" (UID: "f994ce54-fa92-4360-8e3a-4e332218e556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.263944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.262956 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f994ce54-fa92-4360-8e3a-4e332218e556-istiod-ca-cert" pod "f994ce54-fa92-4360-8e3a-4e332218e556" (UID: "f994ce54-fa92-4360-8e3a-4e332218e556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.269058 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-podinfo\") pod \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.269094 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-socket\") pod \"f994ce54-fa92-4360-8e3a-4e332218e556\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.269158 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-data\") pod \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.269187 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-kube-api-access-bcj7q\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-kube-api-access-bcj7q\") pod \"f994ce54-fa92-4360-8e3a-4e332218e556\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.269234 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-credential-socket\") pod \"f994ce54-fa92-4360-8e3a-4e332218e556\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.269263 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-xks-ca\") pod \"58565c64-04fa-4507-b9c7-058493f53a38\" (UID: \"58565c64-04fa-4507-b9c7-058493f53a38\") " Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.269300 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istiod-ca-cert\") pod \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.269334 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-envoy\") pod \"f994ce54-fa92-4360-8e3a-4e332218e556\" (UID: \"f994ce54-fa92-4360-8e3a-4e332218e556\") " Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.269397 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-credential-socket\") pod \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\" (UID: \"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa\") " Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.270497 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.270521 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.270566 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.270581 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9kbs\" (UniqueName: \"kubernetes.io/projected/58565c64-04fa-4507-b9c7-058493f53a38-kube-api-access-b9kbs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.270604 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.270618 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.270632 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.270645 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f994ce54-fa92-4360-8e3a-4e332218e556-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.271201 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-socket" pod "f994ce54-fa92-4360-8e3a-4e332218e556" (UID: "f994ce54-fa92-4360-8e3a-4e332218e556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.273683 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-credential-socket" pod "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" (UID: "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.281636 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-data" pod "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" (UID: "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.281869 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-credential-socket" pod "f994ce54-fa92-4360-8e3a-4e332218e556" (UID: "f994ce54-fa92-4360-8e3a-4e332218e556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.282184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.282175 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istiod-ca-cert" pod "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" (UID: "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:21.292642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.291070 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-kube-api-access-5rb2x" pod "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" (UID: "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa"). InnerVolumeSpecName "kube-api-access-5rb2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.292642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.291252 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:44:21.294694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.294668 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-envoy" pod "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" (UID: "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.294832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.294740 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-envoy" pod "f994ce54-fa92-4360-8e3a-4e332218e556" (UID: "f994ce54-fa92-4360-8e3a-4e332218e556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.295058 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.295006 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-token" pod "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" (UID: "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.302459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.299360 6287 generic.go:356] "Generic (PLEG): container finished" podID="dd434389-e042-41e4-a84f-325058069d49" containerID="b665e428280f3684fcf032d352586a1f2126a289089417df448d1c6d1b555a5f" exitCode=0 Nov 01 22:44:21.302459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.299549 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" event={"ID":"dd434389-e042-41e4-a84f-325058069d49","Type":"ContainerDied","Data":"b665e428280f3684fcf032d352586a1f2126a289089417df448d1c6d1b555a5f"} Nov 01 22:44:21.302459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.299593 6287 scope.go:122] "RemoveContainer" containerID="d6db61681bcf311ee0144282ca573441c65c9a49195ad1757bb7cb832fff93c4" Nov 01 22:44:21.302459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.302306 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f994ce54-fa92-4360-8e3a-4e332218e556-istio-podinfo" pod "f994ce54-fa92-4360-8e3a-4e332218e556" (UID: "f994ce54-fa92-4360-8e3a-4e332218e556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:21.302459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.302354 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" event={"ID":"58565c64-04fa-4507-b9c7-058493f53a38","Type":"ContainerDied","Data":"5e4b29e6ad89aa379c0aa698076ce992adf2b76a3cdec96ad47bfbe4a1b22580"} Nov 01 22:44:21.302459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.302382 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz" Nov 01 22:44:21.302459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.302399 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e4b29e6ad89aa379c0aa698076ce992adf2b76a3cdec96ad47bfbe4a1b22580" Nov 01 22:44:21.304239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.303974 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-podinfo" pod "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" (UID: "61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:21.308441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.308268 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-xks-ca" pod "58565c64-04fa-4507-b9c7-058493f53a38" (UID: "58565c64-04fa-4507-b9c7-058493f53a38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:21.308441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.308323 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-kube-api-access-bcj7q" pod "f994ce54-fa92-4360-8e3a-4e332218e556" (UID: "f994ce54-fa92-4360-8e3a-4e332218e556"). InnerVolumeSpecName "kube-api-access-bcj7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.309053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.309024 6287 generic.go:356] "Generic (PLEG): container finished" podID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerID="9f7e2056f648d663d49db9b855050a3c04096b5fb56fe55daccd38f223f6caff" exitCode=0 Nov 01 22:44:21.309194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.309166 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" event={"ID":"0d3bf83f-b364-4622-975c-f79c1f7cd3c0","Type":"ContainerDied","Data":"9f7e2056f648d663d49db9b855050a3c04096b5fb56fe55daccd38f223f6caff"} Nov 01 22:44:21.311148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.311019 6287 generic.go:356] "Generic (PLEG): container finished" podID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerID="1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f" exitCode=0 Nov 01 22:44:21.311148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.311133 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" event={"ID":"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe","Type":"ContainerDied","Data":"1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f"} Nov 01 22:44:21.311267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.311174 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" event={"ID":"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe","Type":"ContainerDied","Data":"8c735938d8caa258d928b95b9dbf6b57255d05b4da082916429bf265180ba4f9"} Nov 01 22:44:21.311267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.311258 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x" Nov 01 22:44:21.314176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.313905 6287 generic.go:356] "Generic (PLEG): container finished" podID="f994ce54-fa92-4360-8e3a-4e332218e556" containerID="a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c" exitCode=0 Nov 01 22:44:21.314176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.313942 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" Nov 01 22:44:21.314176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.314067 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" event={"ID":"f994ce54-fa92-4360-8e3a-4e332218e556","Type":"ContainerDied","Data":"a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c"} Nov 01 22:44:21.314176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.314124 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9" event={"ID":"f994ce54-fa92-4360-8e3a-4e332218e556","Type":"ContainerDied","Data":"9ac409c18d63fa03fab1e3abc2b7a32eaf68702ee8f7e220fc45d89729f0c79c"} Nov 01 22:44:21.369142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.368296 6287 scope.go:122] "RemoveContainer" containerID="1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f" Nov 01 22:44:21.369142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.368600 6287 generic.go:356] "Generic (PLEG): container finished" podID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerID="3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e" exitCode=0 Nov 01 22:44:21.369142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.368716 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" event={"ID":"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa","Type":"ContainerDied","Data":"3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e"} Nov 01 22:44:21.369142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.368766 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" event={"ID":"61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa","Type":"ContainerDied","Data":"1d544c3c43a7c1b201e394b88df70b2271d0654febf22576c79c355bc034e78a"} Nov 01 22:44:21.369142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.368843 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4" Nov 01 22:44:21.370622 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.370571 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9"] Nov 01 22:44:21.372722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.372617 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-kube-api-access-8clxn\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-kube-api-access-8clxn\") pod \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " Nov 01 22:44:21.372799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.372777 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-token\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-token\") pod \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " Nov 01 22:44:21.372866 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.372825 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istiod-ca-cert\") pod \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " Nov 01 22:44:21.372928 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.372875 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-envoy\") pod \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " Nov 01 22:44:21.373038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.373019 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-certs\") pod \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " Nov 01 22:44:21.373102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.373076 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-data\") pod \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " Nov 01 22:44:21.373214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.373107 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-podinfo\") pod \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " Nov 01 22:44:21.373272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.373210 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-socket\") pod \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " Nov 01 22:44:21.373337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.373270 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-credential-socket\") pod \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\" (UID: \"d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe\") " Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379392 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rb2x\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-kube-api-access-5rb2x\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379421 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379436 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f994ce54-fa92-4360-8e3a-4e332218e556-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379449 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379462 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379476 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379489 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379504 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcj7q\" (UniqueName: \"kubernetes.io/projected/f994ce54-fa92-4360-8e3a-4e332218e556-kube-api-access-bcj7q\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379518 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379531 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58565c64-04fa-4507-b9c7-058493f53a38-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379546 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379559 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f994ce54-fa92-4360-8e3a-4e332218e556-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.379839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.379572 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.380790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.380587 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-credential-socket" pod "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" (UID: "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.381409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.381216 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-certs" pod "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" (UID: "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.381773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.381529 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-data" pod "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" (UID: "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.382736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.382470 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-socket" pod "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" (UID: "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.383717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.383533 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-5n2g9"] Nov 01 22:44:21.387203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.386474 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istiod-ca-cert" pod "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" (UID: "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:21.387567 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.387523 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-token" pod "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" (UID: "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.389869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.388757 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-kube-api-access-8clxn" pod "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" (UID: "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe"). InnerVolumeSpecName "kube-api-access-8clxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.389961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.389873 6287 generic.go:356] "Generic (PLEG): container finished" podID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerID="53d5a8711670c52bcb680f0f13dc3f936f8d7414d17a1766054f0c9bc35f3c52" exitCode=0 Nov 01 22:44:21.389961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.389928 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" event={"ID":"fdff2728-08a8-4ebe-93bf-e240c7639d50","Type":"ContainerDied","Data":"53d5a8711670c52bcb680f0f13dc3f936f8d7414d17a1766054f0c9bc35f3c52"} Nov 01 22:44:21.393877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.393791 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-podinfo" pod "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" (UID: "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:21.394552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.394522 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:44:21.408248 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.408223 6287 scope.go:122] "RemoveContainer" containerID="1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f" Nov 01 22:44:21.408941 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:21.408767 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f\": container with ID starting with 1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f not found: ID does not exist" containerID="1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f" Nov 01 22:44:21.409506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.409480 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f"} err="failed to get container status \"1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f\": rpc error: code = NotFound desc = could not find container \"1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f\": container with ID starting with 1818db03e22a516d5998ad420f2f45ed984e1fe039c84d8105ebb5081baa346f not found: ID does not exist" Nov 01 22:44:21.409621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.409609 6287 scope.go:122] "RemoveContainer" containerID="a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c" Nov 01 22:44:21.420816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.420747 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-envoy" pod "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" (UID: "d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.448787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.448248 6287 scope.go:122] "RemoveContainer" containerID="a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c" Nov 01 22:44:21.450019 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:21.449990 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c\": container with ID starting with a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c not found: ID does not exist" containerID="a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c" Nov 01 22:44:21.450167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.450031 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c"} err="failed to get container status \"a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c\": rpc error: code = NotFound desc = could not find container \"a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c\": container with ID starting with a4bc33de1cad0eda613f53723167fd0cd397718d703e20955be3be48adac134c not found: ID does not exist" Nov 01 22:44:21.450167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.450056 6287 scope.go:122] "RemoveContainer" containerID="3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e" Nov 01 22:44:21.450463 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.450436 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4"] Nov 01 22:44:21.458537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.458510 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-fn9d4"] Nov 01 22:44:21.467849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.467745 6287 scope.go:122] "RemoveContainer" containerID="3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e" Nov 01 22:44:21.468071 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:21.468027 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e\": container with ID starting with 3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e not found: ID does not exist" containerID="3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e" Nov 01 22:44:21.468217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.468058 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e"} err="failed to get container status \"3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e\": rpc error: code = NotFound desc = could not find container \"3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e\": container with ID starting with 3c93b300bc68c696fed2a5f4bc2e3a6203a5c0f8161bff9b16854c81d5d7886e not found: ID does not exist" Nov 01 22:44:21.477528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.477511 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:21.481313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.480815 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" path="/var/lib/kubelet/pods/61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa/volumes" Nov 01 22:44:21.481313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.480852 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-envoy\") pod \"dd434389-e042-41e4-a84f-325058069d49\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " Nov 01 22:44:21.481313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.480986 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-istio-token\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-istio-token\") pod \"dd434389-e042-41e4-a84f-325058069d49\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " Nov 01 22:44:21.481313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.481016 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-socket\") pod \"dd434389-e042-41e4-a84f-325058069d49\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " Nov 01 22:44:21.481313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.481065 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-kube-api-access-6pshj\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-kube-api-access-6pshj\") pod \"dd434389-e042-41e4-a84f-325058069d49\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " Nov 01 22:44:21.481313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.481106 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-data\") pod \"dd434389-e042-41e4-a84f-325058069d49\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " Nov 01 22:44:21.481313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.481237 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-credential-socket\") pod \"dd434389-e042-41e4-a84f-325058069d49\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.481289 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-certs\") pod \"dd434389-e042-41e4-a84f-325058069d49\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.481580 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" path="/var/lib/kubelet/pods/f994ce54-fa92-4360-8e3a-4e332218e556/volumes" Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.481663 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd434389-e042-41e4-a84f-325058069d49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd434389-e042-41e4-a84f-325058069d49-istiod-ca-cert\") pod \"dd434389-e042-41e4-a84f-325058069d49\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.481803 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd434389-e042-41e4-a84f-325058069d49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd434389-e042-41e4-a84f-325058069d49-istio-podinfo\") pod \"dd434389-e042-41e4-a84f-325058069d49\" (UID: \"dd434389-e042-41e4-a84f-325058069d49\") " Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.482859 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.482881 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.482896 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.482909 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.482945 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.482958 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.482970 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.482984 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.483630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.482998 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8clxn\" (UniqueName: \"kubernetes.io/projected/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe-kube-api-access-8clxn\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.490486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.490225 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-data" pod "dd434389-e042-41e4-a84f-325058069d49" (UID: "dd434389-e042-41e4-a84f-325058069d49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.490884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.490619 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-credential-socket" pod "dd434389-e042-41e4-a84f-325058069d49" (UID: "dd434389-e042-41e4-a84f-325058069d49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.492054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.491891 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-certs" pod "dd434389-e042-41e4-a84f-325058069d49" (UID: "dd434389-e042-41e4-a84f-325058069d49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.493513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.492968 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd434389-e042-41e4-a84f-325058069d49-istiod-ca-cert" pod "dd434389-e042-41e4-a84f-325058069d49" (UID: "dd434389-e042-41e4-a84f-325058069d49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:21.493513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.493171 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-envoy" pod "dd434389-e042-41e4-a84f-325058069d49" (UID: "dd434389-e042-41e4-a84f-325058069d49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.493513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.493477 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-socket" pod "dd434389-e042-41e4-a84f-325058069d49" (UID: "dd434389-e042-41e4-a84f-325058069d49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.499391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.499359 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd434389-e042-41e4-a84f-325058069d49-istio-podinfo" pod "dd434389-e042-41e4-a84f-325058069d49" (UID: "dd434389-e042-41e4-a84f-325058069d49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:21.505870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.504483 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-istio-token" pod "dd434389-e042-41e4-a84f-325058069d49" (UID: "dd434389-e042-41e4-a84f-325058069d49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.512311 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.512282 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-kube-api-access-6pshj" pod "dd434389-e042-41e4-a84f-325058069d49" (UID: "dd434389-e042-41e4-a84f-325058069d49"). InnerVolumeSpecName "kube-api-access-6pshj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.532386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.529308 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:44:21.545590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.545565 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:44:21.599090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.599040 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-data\") pod \"fdff2728-08a8-4ebe-93bf-e240c7639d50\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " Nov 01 22:44:21.599274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.599176 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-podinfo\") pod \"fdff2728-08a8-4ebe-93bf-e240c7639d50\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " Nov 01 22:44:21.599343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.599299 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-token\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-token\") pod \"fdff2728-08a8-4ebe-93bf-e240c7639d50\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " Nov 01 22:44:21.599407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.599381 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-kube-api-access-q886l\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-kube-api-access-q886l\") pod \"fdff2728-08a8-4ebe-93bf-e240c7639d50\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " Nov 01 22:44:21.599499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.599480 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdff2728-08a8-4ebe-93bf-e240c7639d50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff2728-08a8-4ebe-93bf-e240c7639d50-istiod-ca-cert\") pod \"fdff2728-08a8-4ebe-93bf-e240c7639d50\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " Nov 01 22:44:21.599694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.599641 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-socket\") pod \"fdff2728-08a8-4ebe-93bf-e240c7639d50\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " Nov 01 22:44:21.599694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.599681 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-envoy\") pod \"fdff2728-08a8-4ebe-93bf-e240c7639d50\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " Nov 01 22:44:21.599822 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.599798 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-credential-socket\") pod \"fdff2728-08a8-4ebe-93bf-e240c7639d50\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " Nov 01 22:44:21.599884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.599837 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-certs\") pod \"fdff2728-08a8-4ebe-93bf-e240c7639d50\" (UID: \"fdff2728-08a8-4ebe-93bf-e240c7639d50\") " Nov 01 22:44:21.601619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.601568 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd434389-e042-41e4-a84f-325058069d49-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.601619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.601611 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.601767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.601627 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.601767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.601642 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.601767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.601658 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pshj\" (UniqueName: \"kubernetes.io/projected/dd434389-e042-41e4-a84f-325058069d49-kube-api-access-6pshj\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.601767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.601677 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.601767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.601692 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.601767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.601707 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd434389-e042-41e4-a84f-325058069d49-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.601767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.601720 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd434389-e042-41e4-a84f-325058069d49-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.603406 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.603375 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-certs" pod "fdff2728-08a8-4ebe-93bf-e240c7639d50" (UID: "fdff2728-08a8-4ebe-93bf-e240c7639d50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.603613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.603588 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-data" pod "fdff2728-08a8-4ebe-93bf-e240c7639d50" (UID: "fdff2728-08a8-4ebe-93bf-e240c7639d50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.605783 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.605757 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdff2728-08a8-4ebe-93bf-e240c7639d50-istiod-ca-cert" pod "fdff2728-08a8-4ebe-93bf-e240c7639d50" (UID: "fdff2728-08a8-4ebe-93bf-e240c7639d50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:21.609831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.609797 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-envoy" pod "fdff2728-08a8-4ebe-93bf-e240c7639d50" (UID: "fdff2728-08a8-4ebe-93bf-e240c7639d50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.610034 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.610004 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-socket" pod "fdff2728-08a8-4ebe-93bf-e240c7639d50" (UID: "fdff2728-08a8-4ebe-93bf-e240c7639d50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.617223 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.617190 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-credential-socket" pod "fdff2728-08a8-4ebe-93bf-e240c7639d50" (UID: "fdff2728-08a8-4ebe-93bf-e240c7639d50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.633450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.633421 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-token" pod "fdff2728-08a8-4ebe-93bf-e240c7639d50" (UID: "fdff2728-08a8-4ebe-93bf-e240c7639d50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.634243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.634218 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-podinfo" pod "fdff2728-08a8-4ebe-93bf-e240c7639d50" (UID: "fdff2728-08a8-4ebe-93bf-e240c7639d50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:21.634803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.634782 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-kube-api-access-q886l" pod "fdff2728-08a8-4ebe-93bf-e240c7639d50" (UID: "fdff2728-08a8-4ebe-93bf-e240c7639d50"). InnerVolumeSpecName "kube-api-access-q886l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.653431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.653404 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x"] Nov 01 22:44:21.660126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.660091 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-twj6x"] Nov 01 22:44:21.702774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.702591 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-token\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-token\") pod \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " Nov 01 22:44:21.702774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.702678 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-credential-socket\") pod \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " Nov 01 22:44:21.702997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.702779 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istiod-ca-cert\") pod \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " Nov 01 22:44:21.702997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.702904 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-kube-api-access-x8gg6\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-kube-api-access-x8gg6\") pod \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " Nov 01 22:44:21.702997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.702977 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-data\") pod \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " Nov 01 22:44:21.703257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.703012 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-envoy\") pod \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " Nov 01 22:44:21.703257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.703091 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-socket\") pod \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " Nov 01 22:44:21.703257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.703183 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-certs\") pod \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " Nov 01 22:44:21.703257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.703239 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-podinfo\") pod \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\" (UID: \"0d3bf83f-b364-4622-975c-f79c1f7cd3c0\") " Nov 01 22:44:21.703473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.703427 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istiod-ca-cert" pod "0d3bf83f-b364-4622-975c-f79c1f7cd3c0" (UID: "0d3bf83f-b364-4622-975c-f79c1f7cd3c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:21.703718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.703695 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-credential-socket" pod "0d3bf83f-b364-4622-975c-f79c1f7cd3c0" (UID: "0d3bf83f-b364-4622-975c-f79c1f7cd3c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.704195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704172 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704301 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704326 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704342 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704356 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704370 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704384 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704399 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q886l\" (UniqueName: \"kubernetes.io/projected/fdff2728-08a8-4ebe-93bf-e240c7639d50-kube-api-access-q886l\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704415 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704429 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff2728-08a8-4ebe-93bf-e240c7639d50-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.704454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.704444 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.705443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.705418 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-socket" pod "0d3bf83f-b364-4622-975c-f79c1f7cd3c0" (UID: "0d3bf83f-b364-4622-975c-f79c1f7cd3c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.705570 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.705541 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-data" pod "0d3bf83f-b364-4622-975c-f79c1f7cd3c0" (UID: "0d3bf83f-b364-4622-975c-f79c1f7cd3c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.705763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.705741 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-certs" pod "0d3bf83f-b364-4622-975c-f79c1f7cd3c0" (UID: "0d3bf83f-b364-4622-975c-f79c1f7cd3c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.706581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.706558 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-podinfo" pod "0d3bf83f-b364-4622-975c-f79c1f7cd3c0" (UID: "0d3bf83f-b364-4622-975c-f79c1f7cd3c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:21.709327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.709300 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-token" pod "0d3bf83f-b364-4622-975c-f79c1f7cd3c0" (UID: "0d3bf83f-b364-4622-975c-f79c1f7cd3c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.709415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.709348 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-kube-api-access-x8gg6" pod "0d3bf83f-b364-4622-975c-f79c1f7cd3c0" (UID: "0d3bf83f-b364-4622-975c-f79c1f7cd3c0"). InnerVolumeSpecName "kube-api-access-x8gg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:21.709584 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.709562 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-envoy" pod "0d3bf83f-b364-4622-975c-f79c1f7cd3c0" (UID: "0d3bf83f-b364-4622-975c-f79c1f7cd3c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:21.806573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.806545 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8gg6\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-kube-api-access-x8gg6\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.806573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.806573 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.806739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.806584 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.806739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.806592 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.806739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.806601 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.806739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.806611 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.806739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.806619 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3bf83f-b364-4622-975c-f79c1f7cd3c0-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:21.820242 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:44:21.820215 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdb3af7_1993_422e_b4f3_d84bfd3e4ac0.slice/crio-1b485ffc41f2b473e781fa51b220558ddce64f48a7e33e3137fbd1c7f692694f WatchSource:0}: Error finding container 1b485ffc41f2b473e781fa51b220558ddce64f48a7e33e3137fbd1c7f692694f: Status 404 returned error can't find the container with id 1b485ffc41f2b473e781fa51b220558ddce64f48a7e33e3137fbd1c7f692694f Nov 01 22:44:21.820916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:21.820893 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj"] Nov 01 22:44:22.402223 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.402143 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" Nov 01 22:44:22.402223 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.402157 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94" event={"ID":"fdff2728-08a8-4ebe-93bf-e240c7639d50","Type":"ContainerDied","Data":"ef9aaa8e4d2152bd0d49e54041b6d960c79288c6230befb7309a18a3d872f738"} Nov 01 22:44:22.402690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.402230 6287 scope.go:122] "RemoveContainer" containerID="53d5a8711670c52bcb680f0f13dc3f936f8d7414d17a1766054f0c9bc35f3c52" Nov 01 22:44:22.403856 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.403838 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" Nov 01 22:44:22.403930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.403866 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg" event={"ID":"dd434389-e042-41e4-a84f-325058069d49","Type":"ContainerDied","Data":"dd8e4c9ff131a466d8a92d436136e53c09922a153c0867e7f17f3ebc71e0b7bd"} Nov 01 22:44:22.405543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.405475 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" Nov 01 22:44:22.405543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.405508 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs" event={"ID":"0d3bf83f-b364-4622-975c-f79c1f7cd3c0","Type":"ContainerDied","Data":"73a9b7bea0746b84c0991035ec261d1c4752d7859a5fb0f95ee021f6e4972f5a"} Nov 01 22:44:22.408897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.408866 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" event={"ID":"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0","Type":"ContainerStarted","Data":"656dd563d4d7330eaf04d10bbe9cb674c1100e73f3ab06cedf688ab0a3962781"} Nov 01 22:44:22.408988 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.408919 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" event={"ID":"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0","Type":"ContainerStarted","Data":"1b485ffc41f2b473e781fa51b220558ddce64f48a7e33e3137fbd1c7f692694f"} Nov 01 22:44:22.417172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.417044 6287 scope.go:122] "RemoveContainer" containerID="b665e428280f3684fcf032d352586a1f2126a289089417df448d1c6d1b555a5f" Nov 01 22:44:22.429812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.429789 6287 scope.go:122] "RemoveContainer" containerID="9f7e2056f648d663d49db9b855050a3c04096b5fb56fe55daccd38f223f6caff" Nov 01 22:44:22.435084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.435041 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" podStartSLOduration=2.435029626 podStartE2EDuration="2.435029626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:22.430557534 +0000 UTC m=+1275.079056768" watchObservedRunningTime="2026-11-01 22:44:22.435029626 +0000 UTC m=+1275.083528861" Nov 01 22:44:22.448568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.448545 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs"] Nov 01 22:44:22.453539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.453517 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-c5ccs"] Nov 01 22:44:22.463819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.463798 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94"] Nov 01 22:44:22.469013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.468990 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-6jz94"] Nov 01 22:44:22.479564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.479543 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg"] Nov 01 22:44:22.484277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.484258 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1d4be-openshift-default-85f677f455-kdzqg"] Nov 01 22:44:22.683806 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.683702 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2"] Nov 01 22:44:22.684079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:22.683989 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" podUID="2caef5a6-ee83-4304-b8be-583b397fb350" containerName="maas-api" containerID="cri-o://16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c" gracePeriod=30 Nov 01 22:44:23.127943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.127860 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj"] Nov 01 22:44:23.138418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.138374 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8bd2b4f-cx7rj"] Nov 01 22:44:23.258937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.258909 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:44:23.355654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.355620 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58"] Nov 01 22:44:23.360684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.360653 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-425870c1-l8f58"] Nov 01 22:44:23.367616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.367570 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" containerID="cri-o://0098098fbfa47ed552b9089cb3f4a5b331a9742b0f0416795644b5bf74b3c2c0" gracePeriod=25 Nov 01 22:44:23.391774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.391739 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" podUID="85879320-4b31-4f5e-a902-797e6f510711" containerName="payload-pre-processing" containerID="cri-o://ece38054aef0b497bca4917b4b5c7169e1875f75c49cc825e88d1ef2669e5265" gracePeriod=25 Nov 01 22:44:23.419011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.418980 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-maas-api-tls\") pod \"2caef5a6-ee83-4304-b8be-583b397fb350\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " Nov 01 22:44:23.419455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.419049 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2caef5a6-ee83-4304-b8be-583b397fb350-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caef5a6-ee83-4304-b8be-583b397fb350-openshift-service-ca\") pod \"2caef5a6-ee83-4304-b8be-583b397fb350\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " Nov 01 22:44:23.419455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.419217 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2caef5a6-ee83-4304-b8be-583b397fb350-kube-api-access-rm655\" (UniqueName: \"kubernetes.io/projected/2caef5a6-ee83-4304-b8be-583b397fb350-kube-api-access-rm655\") pod \"2caef5a6-ee83-4304-b8be-583b397fb350\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " Nov 01 22:44:23.419455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.419261 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-metrics-certs\") pod \"2caef5a6-ee83-4304-b8be-583b397fb350\" (UID: \"2caef5a6-ee83-4304-b8be-583b397fb350\") " Nov 01 22:44:23.419622 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.419463 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2caef5a6-ee83-4304-b8be-583b397fb350-openshift-service-ca" pod "2caef5a6-ee83-4304-b8be-583b397fb350" (UID: "2caef5a6-ee83-4304-b8be-583b397fb350"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:23.420089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.420065 6287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caef5a6-ee83-4304-b8be-583b397fb350-openshift-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:23.420693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.420667 6287 generic.go:356] "Generic (PLEG): container finished" podID="dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0" containerID="656dd563d4d7330eaf04d10bbe9cb674c1100e73f3ab06cedf688ab0a3962781" exitCode=0 Nov 01 22:44:23.420788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.420755 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" event={"ID":"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0","Type":"ContainerDied","Data":"656dd563d4d7330eaf04d10bbe9cb674c1100e73f3ab06cedf688ab0a3962781"} Nov 01 22:44:23.421529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.421506 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-metrics-certs" pod "2caef5a6-ee83-4304-b8be-583b397fb350" (UID: "2caef5a6-ee83-4304-b8be-583b397fb350"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:23.421660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.421640 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-maas-api-tls" pod "2caef5a6-ee83-4304-b8be-583b397fb350" (UID: "2caef5a6-ee83-4304-b8be-583b397fb350"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:23.421858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.421833 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2caef5a6-ee83-4304-b8be-583b397fb350-kube-api-access-rm655" pod "2caef5a6-ee83-4304-b8be-583b397fb350" (UID: "2caef5a6-ee83-4304-b8be-583b397fb350"). InnerVolumeSpecName "kube-api-access-rm655". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:23.423584 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.423560 6287 generic.go:356] "Generic (PLEG): container finished" podID="2caef5a6-ee83-4304-b8be-583b397fb350" containerID="16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c" exitCode=0 Nov 01 22:44:23.424102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.423596 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" Nov 01 22:44:23.424102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.423650 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" event={"ID":"2caef5a6-ee83-4304-b8be-583b397fb350","Type":"ContainerDied","Data":"16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c"} Nov 01 22:44:23.424102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.423695 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2" event={"ID":"2caef5a6-ee83-4304-b8be-583b397fb350","Type":"ContainerDied","Data":"36f47546981a0cff61e39e7e75ec289bd1c3e82d354b5f5637b2412fa1a17a0d"} Nov 01 22:44:23.424102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.423725 6287 scope.go:122] "RemoveContainer" containerID="16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c" Nov 01 22:44:23.446393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.446375 6287 scope.go:122] "RemoveContainer" containerID="16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c" Nov 01 22:44:23.446663 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:23.446636 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c\": container with ID starting with 16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c not found: ID does not exist" containerID="16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c" Nov 01 22:44:23.446705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.446672 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c"} err="failed to get container status \"16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c\": rpc error: code = NotFound desc = could not find container \"16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c\": container with ID starting with 16935bfebbcca5f7965f2c34b2a647b5479455ae2a4d8bdb38191ed8442f2d0c not found: ID does not exist" Nov 01 22:44:23.505296 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.505154 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" path="/var/lib/kubelet/pods/0d3bf83f-b364-4622-975c-f79c1f7cd3c0/volumes" Nov 01 22:44:23.506752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.505938 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13310bb1-f6f9-4bcf-a2e8-6b66e6fad162" path="/var/lib/kubelet/pods/13310bb1-f6f9-4bcf-a2e8-6b66e6fad162/volumes" Nov 01 22:44:23.507202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.507173 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b81c0177-5394-4327-9705-73fc8b1ac2ca" path="/var/lib/kubelet/pods/b81c0177-5394-4327-9705-73fc8b1ac2ca/volumes" Nov 01 22:44:23.508373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.508350 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" path="/var/lib/kubelet/pods/d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe/volumes" Nov 01 22:44:23.519134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.509030 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd434389-e042-41e4-a84f-325058069d49" path="/var/lib/kubelet/pods/dd434389-e042-41e4-a84f-325058069d49/volumes" Nov 01 22:44:23.519134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.509733 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" path="/var/lib/kubelet/pods/fdff2728-08a8-4ebe-93bf-e240c7639d50/volumes" Nov 01 22:44:23.528892 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.528843 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm655\" (UniqueName: \"kubernetes.io/projected/2caef5a6-ee83-4304-b8be-583b397fb350-kube-api-access-rm655\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:23.528892 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.528883 6287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-metrics-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:23.529148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.528902 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2caef5a6-ee83-4304-b8be-583b397fb350-maas-api-tls\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:23.576178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.551550 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2"] Nov 01 22:44:23.579515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.579479 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-425870c1-59bf9c4b78-92zm2"] Nov 01 22:44:23.801308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:23.801217 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:44:24.632163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.632130 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:24.748064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.748029 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-kube-api-access-28r9p\" (UniqueName: \"kubernetes.io/projected/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-kube-api-access-28r9p\") pod \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " Nov 01 22:44:24.748381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.748357 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-xks-ca\") pod \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " Nov 01 22:44:24.748523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.748507 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-openshift-ca\") pod \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\" (UID: \"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0\") " Nov 01 22:44:24.748940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.748911 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-openshift-ca" pod "dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0" (UID: "dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:24.749096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.749079 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:24.750548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.750515 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-xks-ca" pod "dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0" (UID: "dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:24.750711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.750690 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-kube-api-access-28r9p" pod "dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0" (UID: "dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0"). InnerVolumeSpecName "kube-api-access-28r9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:24.852401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.851103 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28r9p\" (UniqueName: \"kubernetes.io/projected/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-kube-api-access-28r9p\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:24.852401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.851160 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:24.945983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.945946 6287 patch_prober.go:29] interesting pod/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:44:24.946187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:24.946008 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:44:25.451286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:25.451255 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" event={"ID":"dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0","Type":"ContainerDied","Data":"1b485ffc41f2b473e781fa51b220558ddce64f48a7e33e3137fbd1c7f692694f"} Nov 01 22:44:25.451387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:25.451305 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj" Nov 01 22:44:25.451387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:25.451330 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b485ffc41f2b473e781fa51b220558ddce64f48a7e33e3137fbd1c7f692694f" Nov 01 22:44:25.490090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:25.490060 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2caef5a6-ee83-4304-b8be-583b397fb350" path="/var/lib/kubelet/pods/2caef5a6-ee83-4304-b8be-583b397fb350/volumes" Nov 01 22:44:26.052769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.052720 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:44:26.111749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.111710 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-kube-api-access-vfp54\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-kube-api-access-vfp54\") pod \"01b573f6-ec63-4be4-85bb-8db796aa13ba\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " Nov 01 22:44:26.111915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.111834 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-socket\") pod \"01b573f6-ec63-4be4-85bb-8db796aa13ba\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " Nov 01 22:44:26.111915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.111901 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-certs\") pod \"01b573f6-ec63-4be4-85bb-8db796aa13ba\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " Nov 01 22:44:26.113022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.112053 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-podinfo\") pod \"01b573f6-ec63-4be4-85bb-8db796aa13ba\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " Nov 01 22:44:26.113022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.112100 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-token\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-token\") pod \"01b573f6-ec63-4be4-85bb-8db796aa13ba\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " Nov 01 22:44:26.113022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.112149 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-credential-socket\") pod \"01b573f6-ec63-4be4-85bb-8db796aa13ba\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " Nov 01 22:44:26.113022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.112179 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-envoy\") pod \"01b573f6-ec63-4be4-85bb-8db796aa13ba\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " Nov 01 22:44:26.113022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.112411 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-socket" pod "01b573f6-ec63-4be4-85bb-8db796aa13ba" (UID: "01b573f6-ec63-4be4-85bb-8db796aa13ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.113022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.112619 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-credential-socket" pod "01b573f6-ec63-4be4-85bb-8db796aa13ba" (UID: "01b573f6-ec63-4be4-85bb-8db796aa13ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.113673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.112230 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-data\") pod \"01b573f6-ec63-4be4-85bb-8db796aa13ba\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " Nov 01 22:44:26.113673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.113568 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b573f6-ec63-4be4-85bb-8db796aa13ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b573f6-ec63-4be4-85bb-8db796aa13ba-istiod-ca-cert\") pod \"01b573f6-ec63-4be4-85bb-8db796aa13ba\" (UID: \"01b573f6-ec63-4be4-85bb-8db796aa13ba\") " Nov 01 22:44:26.114347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.114318 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.114347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.114344 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.114882 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.114831 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-data" pod "01b573f6-ec63-4be4-85bb-8db796aa13ba" (UID: "01b573f6-ec63-4be4-85bb-8db796aa13ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.115065 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.115034 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-certs" pod "01b573f6-ec63-4be4-85bb-8db796aa13ba" (UID: "01b573f6-ec63-4be4-85bb-8db796aa13ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.118950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.118919 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-envoy" pod "01b573f6-ec63-4be4-85bb-8db796aa13ba" (UID: "01b573f6-ec63-4be4-85bb-8db796aa13ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.119746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.119724 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-token" pod "01b573f6-ec63-4be4-85bb-8db796aa13ba" (UID: "01b573f6-ec63-4be4-85bb-8db796aa13ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:26.119991 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.119974 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-kube-api-access-vfp54" pod "01b573f6-ec63-4be4-85bb-8db796aa13ba" (UID: "01b573f6-ec63-4be4-85bb-8db796aa13ba"). InnerVolumeSpecName "kube-api-access-vfp54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:26.120439 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.120413 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b573f6-ec63-4be4-85bb-8db796aa13ba-istiod-ca-cert" pod "01b573f6-ec63-4be4-85bb-8db796aa13ba" (UID: "01b573f6-ec63-4be4-85bb-8db796aa13ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:26.121513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.120600 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-podinfo" pod "01b573f6-ec63-4be4-85bb-8db796aa13ba" (UID: "01b573f6-ec63-4be4-85bb-8db796aa13ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:26.158532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.158500 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:44:26.215515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.215430 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-envoy\") pod \"419eba00-1830-4c81-8c5d-912f18479177\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " Nov 01 22:44:26.215515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.215495 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/419eba00-1830-4c81-8c5d-912f18479177-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419eba00-1830-4c81-8c5d-912f18479177-istio-podinfo\") pod \"419eba00-1830-4c81-8c5d-912f18479177\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " Nov 01 22:44:26.215755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.215517 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-istio-token\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-istio-token\") pod \"419eba00-1830-4c81-8c5d-912f18479177\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " Nov 01 22:44:26.215755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.215546 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-credential-socket\") pod \"419eba00-1830-4c81-8c5d-912f18479177\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " Nov 01 22:44:26.215755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.215642 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-data\") pod \"419eba00-1830-4c81-8c5d-912f18479177\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " Nov 01 22:44:26.215755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.215662 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-certs\") pod \"419eba00-1830-4c81-8c5d-912f18479177\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " Nov 01 22:44:26.215755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.215738 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-socket\") pod \"419eba00-1830-4c81-8c5d-912f18479177\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " Nov 01 22:44:26.215959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.215821 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-kube-api-access-5tvdq\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-kube-api-access-5tvdq\") pod \"419eba00-1830-4c81-8c5d-912f18479177\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " Nov 01 22:44:26.215959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.215933 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/419eba00-1830-4c81-8c5d-912f18479177-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419eba00-1830-4c81-8c5d-912f18479177-istiod-ca-cert\") pod \"419eba00-1830-4c81-8c5d-912f18479177\" (UID: \"419eba00-1830-4c81-8c5d-912f18479177\") " Nov 01 22:44:26.216662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216436 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-credential-socket" pod "419eba00-1830-4c81-8c5d-912f18479177" (UID: "419eba00-1830-4c81-8c5d-912f18479177"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.216817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216673 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.216817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216693 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.216817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216705 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.216817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216717 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b573f6-ec63-4be4-85bb-8db796aa13ba-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.216817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216730 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfp54\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-kube-api-access-vfp54\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.216817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216746 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b573f6-ec63-4be4-85bb-8db796aa13ba-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.216817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216757 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.216817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216753 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419eba00-1830-4c81-8c5d-912f18479177-istiod-ca-cert" pod "419eba00-1830-4c81-8c5d-912f18479177" (UID: "419eba00-1830-4c81-8c5d-912f18479177"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:26.216817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216768 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b573f6-ec63-4be4-85bb-8db796aa13ba-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.217572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.216996 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-socket" pod "419eba00-1830-4c81-8c5d-912f18479177" (UID: "419eba00-1830-4c81-8c5d-912f18479177"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.217572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.217042 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-data" pod "419eba00-1830-4c81-8c5d-912f18479177" (UID: "419eba00-1830-4c81-8c5d-912f18479177"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.217572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.217167 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-certs" pod "419eba00-1830-4c81-8c5d-912f18479177" (UID: "419eba00-1830-4c81-8c5d-912f18479177"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.220486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.220454 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-kube-api-access-5tvdq" pod "419eba00-1830-4c81-8c5d-912f18479177" (UID: "419eba00-1830-4c81-8c5d-912f18479177"). InnerVolumeSpecName "kube-api-access-5tvdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:26.221645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.221622 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-istio-token" pod "419eba00-1830-4c81-8c5d-912f18479177" (UID: "419eba00-1830-4c81-8c5d-912f18479177"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:26.221758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.221732 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/419eba00-1830-4c81-8c5d-912f18479177-istio-podinfo" pod "419eba00-1830-4c81-8c5d-912f18479177" (UID: "419eba00-1830-4c81-8c5d-912f18479177"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:26.223255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.223231 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-envoy" pod "419eba00-1830-4c81-8c5d-912f18479177" (UID: "419eba00-1830-4c81-8c5d-912f18479177"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:26.322203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.322164 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.322203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.322200 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/419eba00-1830-4c81-8c5d-912f18479177-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.322452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.322214 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.322452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.322227 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.322452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.322241 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.322452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.322253 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/419eba00-1830-4c81-8c5d-912f18479177-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.322452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.322266 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tvdq\" (UniqueName: \"kubernetes.io/projected/419eba00-1830-4c81-8c5d-912f18479177-kube-api-access-5tvdq\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.322452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.322279 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/419eba00-1830-4c81-8c5d-912f18479177-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:26.469208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.467353 6287 generic.go:356] "Generic (PLEG): container finished" podID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerID="1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0" exitCode=0 Nov 01 22:44:26.469208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.467392 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" Nov 01 22:44:26.469208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.467441 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" event={"ID":"01b573f6-ec63-4be4-85bb-8db796aa13ba","Type":"ContainerDied","Data":"1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0"} Nov 01 22:44:26.469208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.467502 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8" event={"ID":"01b573f6-ec63-4be4-85bb-8db796aa13ba","Type":"ContainerDied","Data":"2bebd93aa2e18906ccf863db564227f6a94959cb0306271259d8bdc9e4b89e63"} Nov 01 22:44:26.469208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.467531 6287 scope.go:122] "RemoveContainer" containerID="1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0" Nov 01 22:44:26.474541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.474193 6287 generic.go:356] "Generic (PLEG): container finished" podID="419eba00-1830-4c81-8c5d-912f18479177" containerID="3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189" exitCode=0 Nov 01 22:44:26.474541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.474302 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" event={"ID":"419eba00-1830-4c81-8c5d-912f18479177","Type":"ContainerDied","Data":"3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189"} Nov 01 22:44:26.474541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.474345 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" event={"ID":"419eba00-1830-4c81-8c5d-912f18479177","Type":"ContainerDied","Data":"bed025f915506fff2f5cbcb343acba970644c4eec5b33d9f1e3acfedc5d7da7f"} Nov 01 22:44:26.474541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.474422 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2" Nov 01 22:44:26.491778 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.487901 6287 scope.go:122] "RemoveContainer" containerID="1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0" Nov 01 22:44:26.492021 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:26.491992 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0\": container with ID starting with 1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0 not found: ID does not exist" containerID="1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0" Nov 01 22:44:26.492278 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.492252 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0"} err="failed to get container status \"1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0\": rpc error: code = NotFound desc = could not find container \"1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0\": container with ID starting with 1f516ad100ade56f9732bc698d9f85f2407bb025621f98744ff5fcf3e8fcc4f0 not found: ID does not exist" Nov 01 22:44:26.492479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.492455 6287 scope.go:122] "RemoveContainer" containerID="3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189" Nov 01 22:44:26.519053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.518854 6287 scope.go:122] "RemoveContainer" containerID="92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af" Nov 01 22:44:26.523428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.523398 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2"] Nov 01 22:44:26.529186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.529165 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-tj9p2"] Nov 01 22:44:26.537134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.537100 6287 scope.go:122] "RemoveContainer" containerID="3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189" Nov 01 22:44:26.537479 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:26.537456 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189\": container with ID starting with 3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189 not found: ID does not exist" containerID="3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189" Nov 01 22:44:26.537551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.537485 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189"} err="failed to get container status \"3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189\": rpc error: code = NotFound desc = could not find container \"3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189\": container with ID starting with 3e4548b09090643a8afd764b0d988a724a61b67e71ec11043f3c13c7d82a1189 not found: ID does not exist" Nov 01 22:44:26.537551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.537503 6287 scope.go:122] "RemoveContainer" containerID="92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af" Nov 01 22:44:26.537795 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:26.537778 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af\": container with ID starting with 92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af not found: ID does not exist" containerID="92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af" Nov 01 22:44:26.537858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.537799 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af"} err="failed to get container status \"92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af\": rpc error: code = NotFound desc = could not find container \"92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af\": container with ID starting with 92884a428ca7e530543db7551ddb7dc8538ee1d0e5c5ae87b36c86f3000d49af not found: ID does not exist" Nov 01 22:44:26.539840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.539816 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8"] Nov 01 22:44:26.550219 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:26.550191 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ee8d1a-openshift-default-6bc8466c9c-88vb8"] Nov 01 22:44:27.576326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:27.576275 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" path="/var/lib/kubelet/pods/01b573f6-ec63-4be4-85bb-8db796aa13ba/volumes" Nov 01 22:44:27.578569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:27.578543 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419eba00-1830-4c81-8c5d-912f18479177" path="/var/lib/kubelet/pods/419eba00-1830-4c81-8c5d-912f18479177/volumes" Nov 01 22:44:27.588157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:27.582269 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" containerID="cri-o://e6c168f6f0c05e4f5eb90a292d5a2c1b4da6c9aba26db1e35dbbedf436c331de" gracePeriod=45 Nov 01 22:44:27.588157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:27.582417 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" containerID="cri-o://fb6a47634e7229a6f52025a1044fefc2ececb44cb7ba4b02c20b0d0e384fbb57" gracePeriod=45 Nov 01 22:44:27.588157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:27.582524 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" containerID="cri-o://ffbd80ae6071549cf1ec7e3267a8718caa81f52fada557aa2f094355db606fd2" gracePeriod=45 Nov 01 22:44:27.844039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:27.843825 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused" Nov 01 22:44:27.852132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:27.851994 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": dial tcp 10.133.0.96:8000: connect: connection refused" Nov 01 22:44:27.856131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:27.854829 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": dial tcp 10.133.0.85:8000: connect: connection refused" Nov 01 22:44:28.253826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.253786 6287 scope.go:122] "RemoveContainer" containerID="d044fa9964a3a8efbf77a52e314d46ef1ca04bc471705c6bc1d61c3e06cd0ca8" Nov 01 22:44:28.702678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.702534 6287 scope.go:122] "RemoveContainer" containerID="d32682551faf630690ef077eab43efefcd74c089671cb7f493700332761946af" Nov 01 22:44:28.865260 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.865213 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": dial tcp 10.133.0.96:8000: connect: connection refused" Nov 01 22:44:28.865773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.865660 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:44:28.865967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.865688 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": dial tcp 10.133.0.85:8000: connect: connection refused" Nov 01 22:44:28.867460 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.867141 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused" Nov 01 22:44:28.867702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.867686 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:44:28.867839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.867829 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:44:28.901359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.901313 6287 generic.go:356] "Generic (PLEG): container finished" podID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerID="e6c168f6f0c05e4f5eb90a292d5a2c1b4da6c9aba26db1e35dbbedf436c331de" exitCode=0 Nov 01 22:44:28.905320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:28.901601 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" event={"ID":"2fbf50c1-3598-470d-b539-7ac3dfdf0956","Type":"ContainerDied","Data":"e6c168f6f0c05e4f5eb90a292d5a2c1b4da6c9aba26db1e35dbbedf436c331de"} Nov 01 22:44:29.169056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:29.169015 6287 generic.go:356] "Generic (PLEG): container finished" podID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerID="ffbd80ae6071549cf1ec7e3267a8718caa81f52fada557aa2f094355db606fd2" exitCode=0 Nov 01 22:44:29.169297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:29.169128 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" event={"ID":"1fb4781f-2260-4e03-8701-f60193ff81ce","Type":"ContainerDied","Data":"ffbd80ae6071549cf1ec7e3267a8718caa81f52fada557aa2f094355db606fd2"} Nov 01 22:44:29.360955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:29.360902 6287 generic.go:356] "Generic (PLEG): container finished" podID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerID="fb6a47634e7229a6f52025a1044fefc2ececb44cb7ba4b02c20b0d0e384fbb57" exitCode=0 Nov 01 22:44:29.361283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:29.361256 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" event={"ID":"5bc29f3d-8367-4958-bb4e-a489d5ab620e","Type":"ContainerDied","Data":"fb6a47634e7229a6f52025a1044fefc2ececb44cb7ba4b02c20b0d0e384fbb57"} Nov 01 22:44:29.930292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:29.923760 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:44:29.950679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:29.950640 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:44:29.974374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:29.974295 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:44:30.089910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.089882 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-home\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-home\") pod \"1fb4781f-2260-4e03-8701-f60193ff81ce\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " Nov 01 22:44:30.090028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.089940 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tls-certs\") pod \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " Nov 01 22:44:30.090028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.089964 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc29f3d-8367-4958-bb4e-a489d5ab620e-kube-api-access-l4kl9\" (UniqueName: \"kubernetes.io/projected/5bc29f3d-8367-4958-bb4e-a489d5ab620e-kube-api-access-l4kl9\") pod \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " Nov 01 22:44:30.090028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090008 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tls-certs\") pod \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " Nov 01 22:44:30.090236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090108 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-home\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-home\") pod \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " Nov 01 22:44:30.090236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090170 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-model-cache\") pod \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " Nov 01 22:44:30.090236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090230 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb4781f-2260-4e03-8701-f60193ff81ce-kube-api-access-2c8fd\" (UniqueName: \"kubernetes.io/projected/1fb4781f-2260-4e03-8701-f60193ff81ce-kube-api-access-2c8fd\") pod \"1fb4781f-2260-4e03-8701-f60193ff81ce\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " Nov 01 22:44:30.090396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090252 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-home" pod "1fb4781f-2260-4e03-8701-f60193ff81ce" (UID: "1fb4781f-2260-4e03-8701-f60193ff81ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.090396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090288 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tmp-dir\") pod \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " Nov 01 22:44:30.090511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090489 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tmp-dir" pod "5bc29f3d-8367-4958-bb4e-a489d5ab620e" (UID: "5bc29f3d-8367-4958-bb4e-a489d5ab620e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.090832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090806 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-dshm\") pod \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\" (UID: \"5bc29f3d-8367-4958-bb4e-a489d5ab620e\") " Nov 01 22:44:30.090969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090874 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbf50c1-3598-470d-b539-7ac3dfdf0956-kube-api-access-fqfd2\" (UniqueName: \"kubernetes.io/projected/2fbf50c1-3598-470d-b539-7ac3dfdf0956-kube-api-access-fqfd2\") pod \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " Nov 01 22:44:30.090969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.090949 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb4781f-2260-4e03-8701-f60193ff81ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb4781f-2260-4e03-8701-f60193ff81ce-tls-certs\") pod \"1fb4781f-2260-4e03-8701-f60193ff81ce\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " Nov 01 22:44:30.091084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.091051 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-home" pod "5bc29f3d-8367-4958-bb4e-a489d5ab620e" (UID: "5bc29f3d-8367-4958-bb4e-a489d5ab620e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.092275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.091190 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-home\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-home\") pod \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " Nov 01 22:44:30.092275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.091246 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-model-cache\") pod \"1fb4781f-2260-4e03-8701-f60193ff81ce\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " Nov 01 22:44:30.092275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.091290 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-tmp-dir\") pod \"1fb4781f-2260-4e03-8701-f60193ff81ce\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " Nov 01 22:44:30.092275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.091414 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-model-cache\") pod \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " Nov 01 22:44:30.092275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.091459 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-dshm\") pod \"1fb4781f-2260-4e03-8701-f60193ff81ce\" (UID: \"1fb4781f-2260-4e03-8701-f60193ff81ce\") " Nov 01 22:44:30.092275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.091510 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tmp-dir\") pod \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " Nov 01 22:44:30.092275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.091565 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-dshm\") pod \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\" (UID: \"2fbf50c1-3598-470d-b539-7ac3dfdf0956\") " Nov 01 22:44:30.092275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.092092 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-tmp-dir" pod "1fb4781f-2260-4e03-8701-f60193ff81ce" (UID: "1fb4781f-2260-4e03-8701-f60193ff81ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.092275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.092245 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-home" pod "2fbf50c1-3598-470d-b539-7ac3dfdf0956" (UID: "2fbf50c1-3598-470d-b539-7ac3dfdf0956"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.093279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.092657 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.093279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.092676 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.093279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.092691 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.093279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.092703 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.093279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.092714 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.093279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.093092 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tmp-dir" pod "2fbf50c1-3598-470d-b539-7ac3dfdf0956" (UID: "2fbf50c1-3598-470d-b539-7ac3dfdf0956"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.094967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.094941 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-model-cache" pod "5bc29f3d-8367-4958-bb4e-a489d5ab620e" (UID: "5bc29f3d-8367-4958-bb4e-a489d5ab620e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.095073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.094957 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-model-cache" pod "2fbf50c1-3598-470d-b539-7ac3dfdf0956" (UID: "2fbf50c1-3598-470d-b539-7ac3dfdf0956"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.095396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.095321 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-dshm" pod "5bc29f3d-8367-4958-bb4e-a489d5ab620e" (UID: "5bc29f3d-8367-4958-bb4e-a489d5ab620e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.095475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.095450 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-model-cache" pod "1fb4781f-2260-4e03-8701-f60193ff81ce" (UID: "1fb4781f-2260-4e03-8701-f60193ff81ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.095552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.095525 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tls-certs" pod "5bc29f3d-8367-4958-bb4e-a489d5ab620e" (UID: "5bc29f3d-8367-4958-bb4e-a489d5ab620e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:30.095997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.095707 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc29f3d-8367-4958-bb4e-a489d5ab620e-kube-api-access-l4kl9" pod "5bc29f3d-8367-4958-bb4e-a489d5ab620e" (UID: "5bc29f3d-8367-4958-bb4e-a489d5ab620e"). InnerVolumeSpecName "kube-api-access-l4kl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:30.096547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.096435 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbf50c1-3598-470d-b539-7ac3dfdf0956-kube-api-access-fqfd2" pod "2fbf50c1-3598-470d-b539-7ac3dfdf0956" (UID: "2fbf50c1-3598-470d-b539-7ac3dfdf0956"). InnerVolumeSpecName "kube-api-access-fqfd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:30.096547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.096470 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb4781f-2260-4e03-8701-f60193ff81ce-kube-api-access-2c8fd" pod "1fb4781f-2260-4e03-8701-f60193ff81ce" (UID: "1fb4781f-2260-4e03-8701-f60193ff81ce"). InnerVolumeSpecName "kube-api-access-2c8fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:30.096834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.096810 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tls-certs" pod "2fbf50c1-3598-470d-b539-7ac3dfdf0956" (UID: "2fbf50c1-3598-470d-b539-7ac3dfdf0956"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:30.097221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.097202 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-dshm" pod "1fb4781f-2260-4e03-8701-f60193ff81ce" (UID: "1fb4781f-2260-4e03-8701-f60193ff81ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.097901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.097884 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb4781f-2260-4e03-8701-f60193ff81ce-tls-certs" pod "1fb4781f-2260-4e03-8701-f60193ff81ce" (UID: "1fb4781f-2260-4e03-8701-f60193ff81ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:30.098769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.098751 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-dshm" pod "2fbf50c1-3598-470d-b539-7ac3dfdf0956" (UID: "2fbf50c1-3598-470d-b539-7ac3dfdf0956"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:30.194468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194436 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194463 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfd2\" (UniqueName: \"kubernetes.io/projected/2fbf50c1-3598-470d-b539-7ac3dfdf0956-kube-api-access-fqfd2\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194474 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fb4781f-2260-4e03-8701-f60193ff81ce-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194483 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194491 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194502 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fb4781f-2260-4e03-8701-f60193ff81ce-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194510 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194518 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbf50c1-3598-470d-b539-7ac3dfdf0956-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194525 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbf50c1-3598-470d-b539-7ac3dfdf0956-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194533 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4kl9\" (UniqueName: \"kubernetes.io/projected/5bc29f3d-8367-4958-bb4e-a489d5ab620e-kube-api-access-l4kl9\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194541 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc29f3d-8367-4958-bb4e-a489d5ab620e-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194549 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc29f3d-8367-4958-bb4e-a489d5ab620e-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.194737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.194557 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c8fd\" (UniqueName: \"kubernetes.io/projected/1fb4781f-2260-4e03-8701-f60193ff81ce-kube-api-access-2c8fd\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:30.238952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.238925 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:44:30.382801 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.382770 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" Nov 01 22:44:30.382801 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.382789 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" event={"ID":"1fb4781f-2260-4e03-8701-f60193ff81ce","Type":"ContainerDied","Data":"c27271805f168aa4ae8a4e72fb76bd23a4d42f209d76967cc345b76365e5aac2"} Nov 01 22:44:30.383221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.382852 6287 scope.go:122] "RemoveContainer" containerID="ffbd80ae6071549cf1ec7e3267a8718caa81f52fada557aa2f094355db606fd2" Nov 01 22:44:30.385489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.385461 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" event={"ID":"5bc29f3d-8367-4958-bb4e-a489d5ab620e","Type":"ContainerDied","Data":"73fc6a92be7b79fe13daef69ac34fdded40165f2923d2219045ff724eb0a1e98"} Nov 01 22:44:30.385594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.385467 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" Nov 01 22:44:30.387497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.387467 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" event={"ID":"2fbf50c1-3598-470d-b539-7ac3dfdf0956","Type":"ContainerDied","Data":"22ee6968132baedcef5ad7c9a90bbf7e3e7542350ec874f2686da876214a6fb0"} Nov 01 22:44:30.387640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.387526 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx" Nov 01 22:44:30.396830 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.396812 6287 scope.go:122] "RemoveContainer" containerID="fb6a47634e7229a6f52025a1044fefc2ececb44cb7ba4b02c20b0d0e384fbb57" Nov 01 22:44:30.414767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.414742 6287 scope.go:122] "RemoveContainer" containerID="e6c168f6f0c05e4f5eb90a292d5a2c1b4da6c9aba26db1e35dbbedf436c331de" Nov 01 22:44:30.417508 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.417481 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp"] Nov 01 22:44:30.425245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.425222 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp"] Nov 01 22:44:30.433013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.432989 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx"] Nov 01 22:44:30.437749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.437726 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/premium-w0-e25182-kserve-694cbbc687-lzxmx"] Nov 01 22:44:30.448976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.448948 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h"] Nov 01 22:44:30.454155 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.454102 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h"] Nov 01 22:44:30.845625 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.845588 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-2-w0-e25182-kserve-75c6f87849-s7kfp" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:30.850410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:30.850379 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e25182/distinct-w0-e25182-kserve-96f66848b-b5d2h" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:31.052759 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:31.052709 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" podUID="85879320-4b31-4f5e-a902-797e6f510711" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:44:31.187951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:31.187863 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925"] Nov 01 22:44:31.473834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:31.473745 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" path="/var/lib/kubelet/pods/1fb4781f-2260-4e03-8701-f60193ff81ce/volumes" Nov 01 22:44:31.474398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:31.474375 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" path="/var/lib/kubelet/pods/2fbf50c1-3598-470d-b539-7ac3dfdf0956/volumes" Nov 01 22:44:31.474909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:31.474890 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" path="/var/lib/kubelet/pods/5bc29f3d-8367-4958-bb4e-a489d5ab620e/volumes" Nov 01 22:44:33.800594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:33.800556 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:44:36.189525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.189461 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" podUID="5750b6f4-bd5e-4480-9789-153358e1335b" containerName="payload-processing" containerID="cri-o://2db923b28461ce51515e59bd1627835d6e1598d2e043ff12ac73f04267d67bde" gracePeriod=25 Nov 01 22:44:36.457657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.457551 6287 generic.go:356] "Generic (PLEG): container finished" podID="5750b6f4-bd5e-4480-9789-153358e1335b" containerID="2db923b28461ce51515e59bd1627835d6e1598d2e043ff12ac73f04267d67bde" exitCode=0 Nov 01 22:44:36.457657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.457626 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" event={"ID":"5750b6f4-bd5e-4480-9789-153358e1335b","Type":"ContainerDied","Data":"2db923b28461ce51515e59bd1627835d6e1598d2e043ff12ac73f04267d67bde"} Nov 01 22:44:36.563390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.563353 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:44:36.655864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.655833 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5750b6f4-bd5e-4480-9789-153358e1335b-kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/5750b6f4-bd5e-4480-9789-153358e1335b-kube-api-access-ljltx\") pod \"5750b6f4-bd5e-4480-9789-153358e1335b\" (UID: \"5750b6f4-bd5e-4480-9789-153358e1335b\") " Nov 01 22:44:36.656049 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.655905 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5750b6f4-bd5e-4480-9789-153358e1335b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750b6f4-bd5e-4480-9789-153358e1335b-plugins-config-volume\") pod \"5750b6f4-bd5e-4480-9789-153358e1335b\" (UID: \"5750b6f4-bd5e-4480-9789-153358e1335b\") " Nov 01 22:44:36.656376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.656346 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5750b6f4-bd5e-4480-9789-153358e1335b-plugins-config-volume" pod "5750b6f4-bd5e-4480-9789-153358e1335b" (UID: "5750b6f4-bd5e-4480-9789-153358e1335b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:36.656777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.656759 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5750b6f4-bd5e-4480-9789-153358e1335b-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:36.658098 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.658060 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5750b6f4-bd5e-4480-9789-153358e1335b-kube-api-access-ljltx" pod "5750b6f4-bd5e-4480-9789-153358e1335b" (UID: "5750b6f4-bd5e-4480-9789-153358e1335b"). InnerVolumeSpecName "kube-api-access-ljltx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:36.758291 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:36.758258 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljltx\" (UniqueName: \"kubernetes.io/projected/5750b6f4-bd5e-4480-9789-153358e1335b-kube-api-access-ljltx\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:37.514455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:37.513572 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" Nov 01 22:44:37.528911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:37.528888 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925" event={"ID":"5750b6f4-bd5e-4480-9789-153358e1335b","Type":"ContainerDied","Data":"feb6c85dde5e4e179e3461916e71e7a3487a7d09ab59dc8ee269f97f083545d4"} Nov 01 22:44:37.529027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:37.528923 6287 scope.go:122] "RemoveContainer" containerID="2db923b28461ce51515e59bd1627835d6e1598d2e043ff12ac73f04267d67bde" Nov 01 22:44:37.556819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:37.556782 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925"] Nov 01 22:44:37.562288 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:37.562261 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9b7320-686c46dfb7-mx925"] Nov 01 22:44:38.999758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:38.999722 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l"] Nov 01 22:44:39.001205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001174 6287 cpu_manager.go:414] "containerMap: removing container" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" Nov 01 22:44:39.001349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001219 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" Nov 01 22:44:39.001349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001254 6287 cpu_manager.go:414] "containerMap: removing container" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" Nov 01 22:44:39.001349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001262 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" Nov 01 22:44:39.001349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001290 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0" containerName="revoke-keys" Nov 01 22:44:39.001349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001307 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0" containerName="revoke-keys" Nov 01 22:44:39.001349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001333 6287 cpu_manager.go:414] "containerMap: removing container" podUID="58565c64-04fa-4507-b9c7-058493f53a38" containerName="revoke-keys" Nov 01 22:44:39.001349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001341 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58565c64-04fa-4507-b9c7-058493f53a38" containerName="revoke-keys" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001363 6287 cpu_manager.go:414] "containerMap: removing container" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001371 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001414 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001421 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001450 6287 cpu_manager.go:414] "containerMap: removing container" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001458 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001484 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001493 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001516 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001523 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001544 6287 cpu_manager.go:414] "containerMap: removing container" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001552 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001572 6287 cpu_manager.go:414] "containerMap: removing container" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001579 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001603 6287 cpu_manager.go:414] "containerMap: removing container" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001610 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001641 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001648 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001666 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5750b6f4-bd5e-4480-9789-153358e1335b" containerName="payload-processing" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001673 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5750b6f4-bd5e-4480-9789-153358e1335b" containerName="payload-processing" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001695 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2caef5a6-ee83-4304-b8be-583b397fb350" containerName="maas-api" Nov 01 22:44:39.001711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001703 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2caef5a6-ee83-4304-b8be-583b397fb350" containerName="maas-api" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001751 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.001758 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002241 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc29f3d-8367-4958-bb4e-a489d5ab620e" containerName="main" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002276 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002287 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f994ce54-fa92-4360-8e3a-4e332218e556" containerName="istio-proxy" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002297 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002314 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdff2728-08a8-4ebe-93bf-e240c7639d50" containerName="istio-proxy" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002332 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb4781f-2260-4e03-8701-f60193ff81ce" containerName="main" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002340 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4766692-ca30-4c4f-b7ea-2de8b4f5c9fe" containerName="istio-proxy" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002358 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3bf83f-b364-4622-975c-f79c1f7cd3c0" containerName="istio-proxy" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002376 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5750b6f4-bd5e-4480-9789-153358e1335b" containerName="payload-processing" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002394 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2caef5a6-ee83-4304-b8be-583b397fb350" containerName="maas-api" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002414 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbf50c1-3598-470d-b539-7ac3dfdf0956" containerName="main" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002435 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b573f6-ec63-4be4-85bb-8db796aa13ba" containerName="istio-proxy" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002454 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="61505cdf-6bf8-41dd-9d8e-e9c9f4563eaa" containerName="istio-proxy" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002469 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0" containerName="revoke-keys" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002485 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="58565c64-04fa-4507-b9c7-058493f53a38" containerName="revoke-keys" Nov 01 22:44:39.002756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.002503 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="419eba00-1830-4c81-8c5d-912f18479177" containerName="istio-proxy" Nov 01 22:44:39.021712 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.021676 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.025956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.025828 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:39.025956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.025916 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:44:39.027255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.027228 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp"] Nov 01 22:44:39.029023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.028999 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" Nov 01 22:44:39.029147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.029024 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" Nov 01 22:44:39.029506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.029485 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd434389-e042-41e4-a84f-325058069d49" containerName="istio-proxy" Nov 01 22:44:39.035780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.035762 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.038950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.038926 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l"] Nov 01 22:44:39.046748 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.046724 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp"] Nov 01 22:44:39.125033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.125001 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrm7\" (UniqueName: \"kubernetes.io/projected/81db0289-d89d-4920-b91e-a6a73723d01e-kube-api-access-6hrm7\") pod \"maas-api-revoke-sub-3fd98065-xcmvp\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.125231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.125214 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd98065-xcmvp\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.125379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.125357 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-402f0e44-sp78l\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.125460 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.125395 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd98065-xcmvp\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.125460 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.125432 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsm4\" (UniqueName: \"kubernetes.io/projected/f57a9e61-28ff-423b-815b-52bbca44a23f-kube-api-access-rbsm4\") pod \"maas-api-revoke-sub-402f0e44-sp78l\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.125548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.125528 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-402f0e44-sp78l\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.226761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.226732 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrm7\" (UniqueName: \"kubernetes.io/projected/81db0289-d89d-4920-b91e-a6a73723d01e-kube-api-access-6hrm7\") pod \"maas-api-revoke-sub-3fd98065-xcmvp\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.226909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.226850 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd98065-xcmvp\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.227057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.226920 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-402f0e44-sp78l\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.227057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.226939 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd98065-xcmvp\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.227057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.226960 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsm4\" (UniqueName: \"kubernetes.io/projected/f57a9e61-28ff-423b-815b-52bbca44a23f-kube-api-access-rbsm4\") pod \"maas-api-revoke-sub-402f0e44-sp78l\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.227057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.226979 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-402f0e44-sp78l\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.227673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.227648 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd98065-xcmvp\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.227770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.227711 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-402f0e44-sp78l\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.229280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.229260 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd98065-xcmvp\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.229457 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.229440 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-402f0e44-sp78l\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.235506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.235481 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrm7\" (UniqueName: \"kubernetes.io/projected/81db0289-d89d-4920-b91e-a6a73723d01e-kube-api-access-6hrm7\") pod \"maas-api-revoke-sub-3fd98065-xcmvp\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.235917 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.235901 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsm4\" (UniqueName: \"kubernetes.io/projected/f57a9e61-28ff-423b-815b-52bbca44a23f-kube-api-access-rbsm4\") pod \"maas-api-revoke-sub-402f0e44-sp78l\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.352824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.352733 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:39.359325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.359299 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:39.369571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.369547 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt"] Nov 01 22:44:39.377390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.377365 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.384341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.384306 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt"] Nov 01 22:44:39.486893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.486820 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5750b6f4-bd5e-4480-9789-153358e1335b" path="/var/lib/kubelet/pods/5750b6f4-bd5e-4480-9789-153358e1335b/volumes" Nov 01 22:44:39.534837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.534799 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9fx\" (UniqueName: \"kubernetes.io/projected/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-kube-api-access-mh9fx\") pod \"maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.535001 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.534902 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.535077 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.535023 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.636685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.636582 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9fx\" (UniqueName: \"kubernetes.io/projected/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-kube-api-access-mh9fx\") pod \"maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.636869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.636691 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.636869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.636852 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.637459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.637429 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.639306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.639286 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.644063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.644045 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9fx\" (UniqueName: \"kubernetes.io/projected/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-kube-api-access-mh9fx\") pod \"maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:39.671686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.671661 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l"] Nov 01 22:44:39.672316 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:44:39.672294 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57a9e61_28ff_423b_815b_52bbca44a23f.slice/crio-8c1431d2030ea9f0e5329953db63ba8fb0122ccb4671221e18fe581d34a45269 WatchSource:0}: Error finding container 8c1431d2030ea9f0e5329953db63ba8fb0122ccb4671221e18fe581d34a45269: Status 404 returned error can't find the container with id 8c1431d2030ea9f0e5329953db63ba8fb0122ccb4671221e18fe581d34a45269 Nov 01 22:44:39.695704 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:44:39.695677 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81db0289_d89d_4920_b91e_a6a73723d01e.slice/crio-58ffbaddc0137296b425fd789d1e03b1c6c24553dcc649afb2cefb59a1e8a216 WatchSource:0}: Error finding container 58ffbaddc0137296b425fd789d1e03b1c6c24553dcc649afb2cefb59a1e8a216: Status 404 returned error can't find the container with id 58ffbaddc0137296b425fd789d1e03b1c6c24553dcc649afb2cefb59a1e8a216 Nov 01 22:44:39.696662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.696636 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp"] Nov 01 22:44:39.760761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:39.760738 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:40.149612 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:44:40.149581 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db8a3e5_0cb5_4beb_bc05_a2e6f5e885aa.slice/crio-4661c585d61dba12edbaa62658fad1bcc5a2c30af8ff518870ed52aaf0653d92 WatchSource:0}: Error finding container 4661c585d61dba12edbaa62658fad1bcc5a2c30af8ff518870ed52aaf0653d92: Status 404 returned error can't find the container with id 4661c585d61dba12edbaa62658fad1bcc5a2c30af8ff518870ed52aaf0653d92 Nov 01 22:44:40.149980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:40.149746 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt"] Nov 01 22:44:40.544220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:40.544190 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" event={"ID":"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa","Type":"ContainerStarted","Data":"4661c585d61dba12edbaa62658fad1bcc5a2c30af8ff518870ed52aaf0653d92"} Nov 01 22:44:40.545983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:40.545957 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" event={"ID":"f57a9e61-28ff-423b-815b-52bbca44a23f","Type":"ContainerStarted","Data":"2f52bcbf02ed5c0c4f40ac996e4342f6ee990c58fe4fced0fa5dc35253c89b5e"} Nov 01 22:44:40.546195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:40.546174 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" event={"ID":"f57a9e61-28ff-423b-815b-52bbca44a23f","Type":"ContainerStarted","Data":"8c1431d2030ea9f0e5329953db63ba8fb0122ccb4671221e18fe581d34a45269"} Nov 01 22:44:40.547791 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:40.547767 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" event={"ID":"81db0289-d89d-4920-b91e-a6a73723d01e","Type":"ContainerStarted","Data":"5d49a0ea37b9657868e137ea708b72bfcc3857def875627a49b462c3c3d978e4"} Nov 01 22:44:40.547871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:40.547810 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" event={"ID":"81db0289-d89d-4920-b91e-a6a73723d01e","Type":"ContainerStarted","Data":"58ffbaddc0137296b425fd789d1e03b1c6c24553dcc649afb2cefb59a1e8a216"} Nov 01 22:44:40.567599 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:40.567552 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" podStartSLOduration=2.5675420410000003 podStartE2EDuration="2.567542041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:40.564268937 +0000 UTC m=+1293.212768173" watchObservedRunningTime="2026-11-01 22:44:40.567542041 +0000 UTC m=+1293.216041289" Nov 01 22:44:40.583434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:40.583394 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" podStartSLOduration=2.583382068 podStartE2EDuration="2.583382068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:40.581256103 +0000 UTC m=+1293.229755343" watchObservedRunningTime="2026-11-01 22:44:40.583382068 +0000 UTC m=+1293.231881303" Nov 01 22:44:41.041882 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:41.041847 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" podUID="85879320-4b31-4f5e-a902-797e6f510711" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:44:41.558951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:41.558913 6287 generic.go:356] "Generic (PLEG): container finished" podID="81db0289-d89d-4920-b91e-a6a73723d01e" containerID="5d49a0ea37b9657868e137ea708b72bfcc3857def875627a49b462c3c3d978e4" exitCode=0 Nov 01 22:44:41.559410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:41.559001 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" event={"ID":"81db0289-d89d-4920-b91e-a6a73723d01e","Type":"ContainerDied","Data":"5d49a0ea37b9657868e137ea708b72bfcc3857def875627a49b462c3c3d978e4"} Nov 01 22:44:41.565632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:41.565605 6287 generic.go:356] "Generic (PLEG): container finished" podID="4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa" containerID="b463a29e3fcb46c4f2c3baeb9991f73bac22af06b6d4bf9602d61d16337c92ad" exitCode=0 Nov 01 22:44:41.565790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:41.565692 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" event={"ID":"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa","Type":"ContainerDied","Data":"b463a29e3fcb46c4f2c3baeb9991f73bac22af06b6d4bf9602d61d16337c92ad"} Nov 01 22:44:41.567686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:41.567664 6287 generic.go:356] "Generic (PLEG): container finished" podID="f57a9e61-28ff-423b-815b-52bbca44a23f" containerID="2f52bcbf02ed5c0c4f40ac996e4342f6ee990c58fe4fced0fa5dc35253c89b5e" exitCode=0 Nov 01 22:44:41.568158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:41.567706 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" event={"ID":"f57a9e61-28ff-423b-815b-52bbca44a23f","Type":"ContainerDied","Data":"2f52bcbf02ed5c0c4f40ac996e4342f6ee990c58fe4fced0fa5dc35253c89b5e"} Nov 01 22:44:42.877163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:42.874350 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:42.894564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:42.893734 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-xks-ca\") pod \"81db0289-d89d-4920-b91e-a6a73723d01e\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " Nov 01 22:44:42.894564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:42.893945 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81db0289-d89d-4920-b91e-a6a73723d01e-kube-api-access-6hrm7\" (UniqueName: \"kubernetes.io/projected/81db0289-d89d-4920-b91e-a6a73723d01e-kube-api-access-6hrm7\") pod \"81db0289-d89d-4920-b91e-a6a73723d01e\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " Nov 01 22:44:42.894564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:42.894000 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-openshift-ca\") pod \"81db0289-d89d-4920-b91e-a6a73723d01e\" (UID: \"81db0289-d89d-4920-b91e-a6a73723d01e\") " Nov 01 22:44:42.900787 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:42.900745 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-openshift-ca" pod "81db0289-d89d-4920-b91e-a6a73723d01e" (UID: "81db0289-d89d-4920-b91e-a6a73723d01e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:42.907969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:42.903320 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81db0289-d89d-4920-b91e-a6a73723d01e-kube-api-access-6hrm7" pod "81db0289-d89d-4920-b91e-a6a73723d01e" (UID: "81db0289-d89d-4920-b91e-a6a73723d01e"). InnerVolumeSpecName "kube-api-access-6hrm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:42.917527 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:42.917490 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-xks-ca" pod "81db0289-d89d-4920-b91e-a6a73723d01e" (UID: "81db0289-d89d-4920-b91e-a6a73723d01e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:42.929529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:42.929504 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:42.935486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:42.935457 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:43.001580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.001549 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-xks-ca\") pod \"f57a9e61-28ff-423b-815b-52bbca44a23f\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " Nov 01 22:44:43.001760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.001596 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-openshift-ca\") pod \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " Nov 01 22:44:43.001760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.001645 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-kube-api-access-mh9fx\" (UniqueName: \"kubernetes.io/projected/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-kube-api-access-mh9fx\") pod \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " Nov 01 22:44:43.001760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.001716 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-openshift-ca\") pod \"f57a9e61-28ff-423b-815b-52bbca44a23f\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " Nov 01 22:44:43.001927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.001779 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57a9e61-28ff-423b-815b-52bbca44a23f-kube-api-access-rbsm4\" (UniqueName: \"kubernetes.io/projected/f57a9e61-28ff-423b-815b-52bbca44a23f-kube-api-access-rbsm4\") pod \"f57a9e61-28ff-423b-815b-52bbca44a23f\" (UID: \"f57a9e61-28ff-423b-815b-52bbca44a23f\") " Nov 01 22:44:43.001927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.001869 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-xks-ca\") pod \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\" (UID: \"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa\") " Nov 01 22:44:43.002138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.001990 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-openshift-ca" pod "4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa" (UID: "4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:43.002138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.002102 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-openshift-ca" pod "f57a9e61-28ff-423b-815b-52bbca44a23f" (UID: "f57a9e61-28ff-423b-815b-52bbca44a23f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:43.002801 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.002772 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrm7\" (UniqueName: \"kubernetes.io/projected/81db0289-d89d-4920-b91e-a6a73723d01e-kube-api-access-6hrm7\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.002943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.002803 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.002943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.002820 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.002943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.002836 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81db0289-d89d-4920-b91e-a6a73723d01e-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.002943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.002858 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.003842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.003819 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-xks-ca" pod "f57a9e61-28ff-423b-815b-52bbca44a23f" (UID: "f57a9e61-28ff-423b-815b-52bbca44a23f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:43.004590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.004052 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-xks-ca" pod "4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa" (UID: "4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:43.004590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.004387 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57a9e61-28ff-423b-815b-52bbca44a23f-kube-api-access-rbsm4" pod "f57a9e61-28ff-423b-815b-52bbca44a23f" (UID: "f57a9e61-28ff-423b-815b-52bbca44a23f"). InnerVolumeSpecName "kube-api-access-rbsm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:43.004590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.004425 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-kube-api-access-mh9fx" pod "4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa" (UID: "4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa"). InnerVolumeSpecName "kube-api-access-mh9fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:43.104881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.104800 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.104881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.104828 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f57a9e61-28ff-423b-815b-52bbca44a23f-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.104881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.104838 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh9fx\" (UniqueName: \"kubernetes.io/projected/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa-kube-api-access-mh9fx\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.104881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.104849 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbsm4\" (UniqueName: \"kubernetes.io/projected/f57a9e61-28ff-423b-815b-52bbca44a23f-kube-api-access-rbsm4\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.685020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.682804 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" event={"ID":"81db0289-d89d-4920-b91e-a6a73723d01e","Type":"ContainerDied","Data":"58ffbaddc0137296b425fd789d1e03b1c6c24553dcc649afb2cefb59a1e8a216"} Nov 01 22:44:43.685020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.682876 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58ffbaddc0137296b425fd789d1e03b1c6c24553dcc649afb2cefb59a1e8a216" Nov 01 22:44:43.685020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.682955 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp" Nov 01 22:44:43.690325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.690290 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" event={"ID":"4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa","Type":"ContainerDied","Data":"4661c585d61dba12edbaa62658fad1bcc5a2c30af8ff518870ed52aaf0653d92"} Nov 01 22:44:43.690553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.690534 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4661c585d61dba12edbaa62658fad1bcc5a2c30af8ff518870ed52aaf0653d92" Nov 01 22:44:43.690723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.690708 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt" Nov 01 22:44:43.694913 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.694887 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" event={"ID":"f57a9e61-28ff-423b-815b-52bbca44a23f","Type":"ContainerDied","Data":"8c1431d2030ea9f0e5329953db63ba8fb0122ccb4671221e18fe581d34a45269"} Nov 01 22:44:43.695434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.695284 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c1431d2030ea9f0e5329953db63ba8fb0122ccb4671221e18fe581d34a45269" Nov 01 22:44:43.695434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.695086 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l" Nov 01 22:44:43.698754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.697689 6287 generic.go:356] "Generic (PLEG): container finished" podID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerID="0098098fbfa47ed552b9089cb3f4a5b331a9742b0f0416795644b5bf74b3c2c0" exitCode=0 Nov 01 22:44:43.698754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.697806 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" event={"ID":"55badcc9-0370-42d9-a059-5537ff2acdb6","Type":"ContainerDied","Data":"0098098fbfa47ed552b9089cb3f4a5b331a9742b0f0416795644b5bf74b3c2c0"} Nov 01 22:44:43.701385 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.701193 6287 generic.go:356] "Generic (PLEG): container finished" podID="85879320-4b31-4f5e-a902-797e6f510711" containerID="ece38054aef0b497bca4917b4b5c7169e1875f75c49cc825e88d1ef2669e5265" exitCode=0 Nov 01 22:44:43.701385 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.701226 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" event={"ID":"85879320-4b31-4f5e-a902-797e6f510711","Type":"ContainerDied","Data":"ece38054aef0b497bca4917b4b5c7169e1875f75c49cc825e88d1ef2669e5265"} Nov 01 22:44:43.760174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.760083 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:44:43.818478 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.818447 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:44:43.890886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.890821 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85879320-4b31-4f5e-a902-797e6f510711-kube-api-access-j4xcw\" (UniqueName: \"kubernetes.io/projected/85879320-4b31-4f5e-a902-797e6f510711-kube-api-access-j4xcw\") pod \"85879320-4b31-4f5e-a902-797e6f510711\" (UID: \"85879320-4b31-4f5e-a902-797e6f510711\") " Nov 01 22:44:43.891380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.890949 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55badcc9-0370-42d9-a059-5537ff2acdb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55badcc9-0370-42d9-a059-5537ff2acdb6-plugins-config-volume\") pod \"55badcc9-0370-42d9-a059-5537ff2acdb6\" (UID: \"55badcc9-0370-42d9-a059-5537ff2acdb6\") " Nov 01 22:44:43.891380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.891014 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85879320-4b31-4f5e-a902-797e6f510711-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85879320-4b31-4f5e-a902-797e6f510711-plugins-config-volume\") pod \"85879320-4b31-4f5e-a902-797e6f510711\" (UID: \"85879320-4b31-4f5e-a902-797e6f510711\") " Nov 01 22:44:43.891380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.891318 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55badcc9-0370-42d9-a059-5537ff2acdb6-kube-api-access-l4xg6\" (UniqueName: \"kubernetes.io/projected/55badcc9-0370-42d9-a059-5537ff2acdb6-kube-api-access-l4xg6\") pod \"55badcc9-0370-42d9-a059-5537ff2acdb6\" (UID: \"55badcc9-0370-42d9-a059-5537ff2acdb6\") " Nov 01 22:44:43.892024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.892003 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85879320-4b31-4f5e-a902-797e6f510711-plugins-config-volume" pod "85879320-4b31-4f5e-a902-797e6f510711" (UID: "85879320-4b31-4f5e-a902-797e6f510711"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:43.892228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.891636 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55badcc9-0370-42d9-a059-5537ff2acdb6-plugins-config-volume" pod "55badcc9-0370-42d9-a059-5537ff2acdb6" (UID: "55badcc9-0370-42d9-a059-5537ff2acdb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:43.893631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.893595 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85879320-4b31-4f5e-a902-797e6f510711-kube-api-access-j4xcw" pod "85879320-4b31-4f5e-a902-797e6f510711" (UID: "85879320-4b31-4f5e-a902-797e6f510711"). InnerVolumeSpecName "kube-api-access-j4xcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:43.894802 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.894776 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55badcc9-0370-42d9-a059-5537ff2acdb6-kube-api-access-l4xg6" pod "55badcc9-0370-42d9-a059-5537ff2acdb6" (UID: "55badcc9-0370-42d9-a059-5537ff2acdb6"). InnerVolumeSpecName "kube-api-access-l4xg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:43.994173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.994144 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55badcc9-0370-42d9-a059-5537ff2acdb6-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.994173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.994172 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85879320-4b31-4f5e-a902-797e6f510711-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.994395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.994181 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4xg6\" (UniqueName: \"kubernetes.io/projected/55badcc9-0370-42d9-a059-5537ff2acdb6-kube-api-access-l4xg6\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:43.994395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:43.994189 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xcw\" (UniqueName: \"kubernetes.io/projected/85879320-4b31-4f5e-a902-797e6f510711-kube-api-access-j4xcw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:44.711469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.711436 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" Nov 01 22:44:44.711469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.711443 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" event={"ID":"55badcc9-0370-42d9-a059-5537ff2acdb6","Type":"ContainerDied","Data":"51a91c1fee85c6de96929307c757455a199a1f60e520a46c28a7c31d8418f310"} Nov 01 22:44:44.711723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.711502 6287 scope.go:122] "RemoveContainer" containerID="0098098fbfa47ed552b9089cb3f4a5b331a9742b0f0416795644b5bf74b3c2c0" Nov 01 22:44:44.713038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.713009 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" Nov 01 22:44:44.713174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.713037 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj" event={"ID":"85879320-4b31-4f5e-a902-797e6f510711","Type":"ContainerDied","Data":"2dfabce55f72d205deb0851f57ef80ee32c07e6063b6fa2c2f9d9da0cd43251c"} Nov 01 22:44:44.725878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.725862 6287 scope.go:122] "RemoveContainer" containerID="ece38054aef0b497bca4917b4b5c7169e1875f75c49cc825e88d1ef2669e5265" Nov 01 22:44:44.746139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.746092 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj"] Nov 01 22:44:44.757838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.757808 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e25182-58f5888995-9cbjj"] Nov 01 22:44:44.778044 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.778018 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf"] Nov 01 22:44:44.783317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.783287 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf"] Nov 01 22:44:44.799126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:44.798745 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e25182-6f97589bc7-4wvvf" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.51:50052\" within 1s: context deadline exceeded" Nov 01 22:44:44.799126 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:44:44.798853 6287 logging.go:55] [core] [Channel #3707 SubChannel #3708]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.51:50052", ServerName: "10.133.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.51:50052: operation was canceled" Nov 01 22:44:45.472855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:45.472822 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" path="/var/lib/kubelet/pods/55badcc9-0370-42d9-a059-5537ff2acdb6/volumes" Nov 01 22:44:45.473821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:45.473803 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85879320-4b31-4f5e-a902-797e6f510711" path="/var/lib/kubelet/pods/85879320-4b31-4f5e-a902-797e6f510711/volumes" Nov 01 22:44:45.589470 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:44:45.589431 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:44:48.610172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:48.610139 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj"] Nov 01 22:44:48.615352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:48.615323 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b7320-zt4mj"] Nov 01 22:44:49.472961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:49.472915 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0" path="/var/lib/kubelet/pods/dcdb3af7-1993-422e-b4f3-d84bfd3e4ac0/volumes" Nov 01 22:44:57.769452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:57.769407 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt"] Nov 01 22:44:57.775183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:57.775102 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e25182-2q9qt"] Nov 01 22:44:58.311468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.311428 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh"] Nov 01 22:44:58.312214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312197 6287 cpu_manager.go:414] "containerMap: removing container" podUID="81db0289-d89d-4920-b91e-a6a73723d01e" containerName="revoke-keys" Nov 01 22:44:58.312214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312215 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81db0289-d89d-4920-b91e-a6a73723d01e" containerName="revoke-keys" Nov 01 22:44:58.312342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312233 6287 cpu_manager.go:414] "containerMap: removing container" podUID="85879320-4b31-4f5e-a902-797e6f510711" containerName="payload-pre-processing" Nov 01 22:44:58.312342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312239 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85879320-4b31-4f5e-a902-797e6f510711" containerName="payload-pre-processing" Nov 01 22:44:58.312342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312247 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f57a9e61-28ff-423b-815b-52bbca44a23f" containerName="revoke-keys" Nov 01 22:44:58.312342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312252 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57a9e61-28ff-423b-815b-52bbca44a23f" containerName="revoke-keys" Nov 01 22:44:58.312342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312266 6287 cpu_manager.go:414] "containerMap: removing container" podUID="4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa" containerName="revoke-keys" Nov 01 22:44:58.312342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312271 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa" containerName="revoke-keys" Nov 01 22:44:58.312342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312283 6287 cpu_manager.go:414] "containerMap: removing container" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" Nov 01 22:44:58.312342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312288 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" Nov 01 22:44:58.312592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312398 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="81db0289-d89d-4920-b91e-a6a73723d01e" containerName="revoke-keys" Nov 01 22:44:58.312592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312408 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="85879320-4b31-4f5e-a902-797e6f510711" containerName="payload-pre-processing" Nov 01 22:44:58.312592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312415 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57a9e61-28ff-423b-815b-52bbca44a23f" containerName="revoke-keys" Nov 01 22:44:58.312592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312424 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa" containerName="revoke-keys" Nov 01 22:44:58.312592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.312431 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="55badcc9-0370-42d9-a059-5537ff2acdb6" containerName="payload-processing" Nov 01 22:44:58.316642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.316576 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.325174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.325103 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:58.325569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.325543 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:44:58.327694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.327672 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh"] Nov 01 22:44:58.442681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.442644 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.442887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.442696 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.442887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.442874 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbj6\" (UniqueName: \"kubernetes.io/projected/d2dfd10d-dedb-48e2-a725-1f3efd485f46-kube-api-access-wtbj6\") pod \"maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.544995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.544965 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.545178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.545003 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.545178 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.545084 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbj6\" (UniqueName: \"kubernetes.io/projected/d2dfd10d-dedb-48e2-a725-1f3efd485f46-kube-api-access-wtbj6\") pod \"maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.545761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.545733 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.547495 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.547468 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.553760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.553735 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbj6\" (UniqueName: \"kubernetes.io/projected/d2dfd10d-dedb-48e2-a725-1f3efd485f46-kube-api-access-wtbj6\") pod \"maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.638445 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.638373 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:44:58.980938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:58.980906 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh"] Nov 01 22:44:58.981455 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:44:58.981432 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2dfd10d_dedb_48e2_a725_1f3efd485f46.slice/crio-0b6f139d9916def25caaaefad80b01cc1883d19e6bf2331dd0e553b2cda575bd WatchSource:0}: Error finding container 0b6f139d9916def25caaaefad80b01cc1883d19e6bf2331dd0e553b2cda575bd: Status 404 returned error can't find the container with id 0b6f139d9916def25caaaefad80b01cc1883d19e6bf2331dd0e553b2cda575bd Nov 01 22:44:59.605786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.605490 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f222087-18d6-46b2-9d37-0c4c510d8589" path="/var/lib/kubelet/pods/3f222087-18d6-46b2-9d37-0c4c510d8589/volumes" Nov 01 22:44:59.727008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.726934 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv"] Nov 01 22:44:59.731802 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.731781 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.750299 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.750272 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv"] Nov 01 22:44:59.805170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.805095 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.805313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.805247 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx8rk\" (UniqueName: \"kubernetes.io/projected/cdffff35-fc26-4169-8373-f92a0d756b26-kube-api-access-wx8rk\") pod \"maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.805538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.805513 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.907509 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.907483 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.907634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.907586 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.907705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.907647 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx8rk\" (UniqueName: \"kubernetes.io/projected/cdffff35-fc26-4169-8373-f92a0d756b26-kube-api-access-wx8rk\") pod \"maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.908153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.908091 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.909692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.909672 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.916360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.916338 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx8rk\" (UniqueName: \"kubernetes.io/projected/cdffff35-fc26-4169-8373-f92a0d756b26-kube-api-access-wx8rk\") pod \"maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:44:59.956077 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.956052 6287 generic.go:356] "Generic (PLEG): container finished" podID="d2dfd10d-dedb-48e2-a725-1f3efd485f46" containerID="29f95c319ac41c4515c5ae21d31580658f60c9fe8b01388da54f4c81b1ebe209" exitCode=0 Nov 01 22:44:59.956187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.956143 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" event={"ID":"d2dfd10d-dedb-48e2-a725-1f3efd485f46","Type":"ContainerDied","Data":"29f95c319ac41c4515c5ae21d31580658f60c9fe8b01388da54f4c81b1ebe209"} Nov 01 22:44:59.956233 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:44:59.956193 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" event={"ID":"d2dfd10d-dedb-48e2-a725-1f3efd485f46","Type":"ContainerStarted","Data":"0b6f139d9916def25caaaefad80b01cc1883d19e6bf2331dd0e553b2cda575bd"} Nov 01 22:45:00.053007 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.052983 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:45:00.152840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.152806 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf"] Nov 01 22:45:00.159519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.159492 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:00.162408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.162383 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk"] Nov 01 22:45:00.170043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.167811 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:00.170043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.168592 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb"] Nov 01 22:45:00.175421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.175397 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:00.187290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.178251 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9"] Nov 01 22:45:00.194325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.192010 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf"] Nov 01 22:45:00.194325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.192185 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:00.209208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.208875 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk"] Nov 01 22:45:00.213045 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.213016 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk\" (UID: \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:00.213356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.213331 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtlw4\" (UniqueName: \"kubernetes.io/projected/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-kube-api-access-rtlw4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb\" (UID: \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:00.213557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.213538 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf\" (UID: \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:00.213691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.213678 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkg9k\" (UniqueName: \"kubernetes.io/projected/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-kube-api-access-dkg9k\") pod \"maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk\" (UID: \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:00.213794 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.213785 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rld\" (UniqueName: \"kubernetes.io/projected/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-kube-api-access-n5rld\") pod \"maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf\" (UID: \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:00.213970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.213958 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb\" (UID: \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:00.218190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.218169 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb"] Nov 01 22:45:00.227041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.227017 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9"] Nov 01 22:45:00.246742 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.246717 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5"] Nov 01 22:45:00.251901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.251879 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:00.262786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.262691 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm"] Nov 01 22:45:00.268350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.268331 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr"] Nov 01 22:45:00.268494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.268471 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:00.273194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.273177 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:00.284979 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.282367 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5"] Nov 01 22:45:00.289904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.289876 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm"] Nov 01 22:45:00.316338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.316265 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtlw4\" (UniqueName: \"kubernetes.io/projected/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-kube-api-access-rtlw4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb\" (UID: \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:00.316546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.316524 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c8b3b3-4de9-430a-a5df-ad824446bead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5\" (UID: \"a2c8b3b3-4de9-430a-a5df-ad824446bead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:00.316683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.316666 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxzfn\" (UniqueName: \"kubernetes.io/projected/a2c8b3b3-4de9-430a-a5df-ad824446bead-kube-api-access-dxzfn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5\" (UID: \"a2c8b3b3-4de9-430a-a5df-ad824446bead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:00.316826 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.316811 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjcsn\" (UniqueName: \"kubernetes.io/projected/ca6a9c68-a26b-4648-812b-1287c245cff0-kube-api-access-hjcsn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr\" (UID: \"ca6a9c68-a26b-4648-812b-1287c245cff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:00.317009 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.316990 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf\" (UID: \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:00.317243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.317226 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkg9k\" (UniqueName: \"kubernetes.io/projected/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-kube-api-access-dkg9k\") pod \"maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk\" (UID: \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:00.317462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.317437 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rld\" (UniqueName: \"kubernetes.io/projected/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-kube-api-access-n5rld\") pod \"maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf\" (UID: \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:00.317556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.317540 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab1ac-2a1c-40cf-b39b-19981d199020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9\" (UID: \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:00.317664 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.317645 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6a9c68-a26b-4648-812b-1287c245cff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr\" (UID: \"ca6a9c68-a26b-4648-812b-1287c245cff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:00.317728 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.317684 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb\" (UID: \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:00.317791 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.317762 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6cc\" (UniqueName: \"kubernetes.io/projected/0b7ab1ac-2a1c-40cf-b39b-19981d199020-kube-api-access-lh6cc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9\" (UID: \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:00.317850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.317803 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5w4\" (UniqueName: \"kubernetes.io/projected/eb8e2476-fb3a-496a-9f5a-ba753a927d97-kube-api-access-mx5w4\") pod \"maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm\" (UID: \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:00.317850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.317835 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8e2476-fb3a-496a-9f5a-ba753a927d97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm\" (UID: \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:00.318602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.317897 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk\" (UID: \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:00.318602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.318398 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf\" (UID: \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:00.318715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.318694 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk\" (UID: \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:00.322124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.322081 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb\" (UID: \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:00.327166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.326620 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr"] Nov 01 22:45:00.334221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.334196 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x"] Nov 01 22:45:00.341000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.340973 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:00.349520 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.349252 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rld\" (UniqueName: \"kubernetes.io/projected/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-kube-api-access-n5rld\") pod \"maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf\" (UID: \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:00.354066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.352383 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf"] Nov 01 22:45:00.358213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.358191 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg"] Nov 01 22:45:00.358355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.358208 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:00.364299 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.364272 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x"] Nov 01 22:45:00.364415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.364371 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:00.390838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.389304 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf"] Nov 01 22:45:00.404253 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.404228 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg"] Nov 01 22:45:00.410716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.410694 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkg9k\" (UniqueName: \"kubernetes.io/projected/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-kube-api-access-dkg9k\") pod \"maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk\" (UID: \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:00.435910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.418890 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtlw4\" (UniqueName: \"kubernetes.io/projected/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-kube-api-access-rtlw4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb\" (UID: \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:00.435910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.426513 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv"] Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.474057 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxzfn\" (UniqueName: \"kubernetes.io/projected/a2c8b3b3-4de9-430a-a5df-ad824446bead-kube-api-access-dxzfn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5\" (UID: \"a2c8b3b3-4de9-430a-a5df-ad824446bead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.474132 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmsw6\" (UniqueName: \"kubernetes.io/projected/9045c5cb-a562-4704-8bb9-5c63bf0c4585-kube-api-access-tmsw6\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x\" (UID: \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.474204 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjcsn\" (UniqueName: \"kubernetes.io/projected/ca6a9c68-a26b-4648-812b-1287c245cff0-kube-api-access-hjcsn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr\" (UID: \"ca6a9c68-a26b-4648-812b-1287c245cff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.474309 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0db05bf-2b57-40d9-a059-5532c25fe5e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-hvtgf\" (UID: \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.474417 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4s8s\" (UniqueName: \"kubernetes.io/projected/e0db05bf-2b57-40d9-a059-5532c25fe5e3-kube-api-access-m4s8s\") pod \"maas-api-key-cleanup-29892885-hvtgf\" (UID: \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.474477 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab1ac-2a1c-40cf-b39b-19981d199020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9\" (UID: \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.474555 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77e5165-0098-48e8-8df6-ff4046a1d37b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg\" (UID: \"e77e5165-0098-48e8-8df6-ff4046a1d37b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.474609 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/e77e5165-0098-48e8-8df6-ff4046a1d37b-kube-api-access-6sql4\") pod \"maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg\" (UID: \"e77e5165-0098-48e8-8df6-ff4046a1d37b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.474642 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6a9c68-a26b-4648-812b-1287c245cff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr\" (UID: \"ca6a9c68-a26b-4648-812b-1287c245cff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.475800 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab1ac-2a1c-40cf-b39b-19981d199020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9\" (UID: \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.475955 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045c5cb-a562-4704-8bb9-5c63bf0c4585-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x\" (UID: \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.475985 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6a9c68-a26b-4648-812b-1287c245cff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr\" (UID: \"ca6a9c68-a26b-4648-812b-1287c245cff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.476020 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6cc\" (UniqueName: \"kubernetes.io/projected/0b7ab1ac-2a1c-40cf-b39b-19981d199020-kube-api-access-lh6cc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9\" (UID: \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.476124 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5w4\" (UniqueName: \"kubernetes.io/projected/eb8e2476-fb3a-496a-9f5a-ba753a927d97-kube-api-access-mx5w4\") pod \"maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm\" (UID: \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.476167 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8e2476-fb3a-496a-9f5a-ba753a927d97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm\" (UID: \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:00.476451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.476294 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c8b3b3-4de9-430a-a5df-ad824446bead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5\" (UID: \"a2c8b3b3-4de9-430a-a5df-ad824446bead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:00.477668 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.477140 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c8b3b3-4de9-430a-a5df-ad824446bead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5\" (UID: \"a2c8b3b3-4de9-430a-a5df-ad824446bead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:00.477668 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.477624 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8e2476-fb3a-496a-9f5a-ba753a927d97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm\" (UID: \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:00.483920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.483901 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh"] Nov 01 22:45:00.491149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.490751 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:00.491763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.491741 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6cc\" (UniqueName: \"kubernetes.io/projected/0b7ab1ac-2a1c-40cf-b39b-19981d199020-kube-api-access-lh6cc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9\" (UID: \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:00.491954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.491929 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjcsn\" (UniqueName: \"kubernetes.io/projected/ca6a9c68-a26b-4648-812b-1287c245cff0-kube-api-access-hjcsn\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr\" (UID: \"ca6a9c68-a26b-4648-812b-1287c245cff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:00.492033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.492008 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5w4\" (UniqueName: \"kubernetes.io/projected/eb8e2476-fb3a-496a-9f5a-ba753a927d97-kube-api-access-mx5w4\") pod \"maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm\" (UID: \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:00.495630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.495601 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxzfn\" (UniqueName: \"kubernetes.io/projected/a2c8b3b3-4de9-430a-a5df-ad824446bead-kube-api-access-dxzfn\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5\" (UID: \"a2c8b3b3-4de9-430a-a5df-ad824446bead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:00.497711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.497691 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh"] Nov 01 22:45:00.500323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.499886 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:00.511998 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.511978 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:00.521719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.521140 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:00.529149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.529080 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:00.593937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.593904 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:00.597166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.597097 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.597353 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4s8s\" (UniqueName: \"kubernetes.io/projected/e0db05bf-2b57-40d9-a059-5532c25fe5e3-kube-api-access-m4s8s\") pod \"maas-api-key-cleanup-29892885-hvtgf\" (UID: \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.597459 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77e5165-0098-48e8-8df6-ff4046a1d37b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg\" (UID: \"e77e5165-0098-48e8-8df6-ff4046a1d37b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.597507 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/e77e5165-0098-48e8-8df6-ff4046a1d37b-kube-api-access-6sql4\") pod \"maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg\" (UID: \"e77e5165-0098-48e8-8df6-ff4046a1d37b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.597542 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blm8c\" (UniqueName: \"kubernetes.io/projected/fdcadc29-3748-4c2b-80a2-60d946bca6e1-kube-api-access-blm8c\") pod \"maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh\" (UID: \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.597601 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045c5cb-a562-4704-8bb9-5c63bf0c4585-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x\" (UID: \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.597641 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcadc29-3748-4c2b-80a2-60d946bca6e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh\" (UID: \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.597792 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmsw6\" (UniqueName: \"kubernetes.io/projected/9045c5cb-a562-4704-8bb9-5c63bf0c4585-kube-api-access-tmsw6\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x\" (UID: \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.597916 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0db05bf-2b57-40d9-a059-5532c25fe5e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-hvtgf\" (UID: \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.599038 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77e5165-0098-48e8-8df6-ff4046a1d37b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg\" (UID: \"e77e5165-0098-48e8-8df6-ff4046a1d37b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.599997 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045c5cb-a562-4704-8bb9-5c63bf0c4585-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x\" (UID: \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:00.601145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.600854 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0db05bf-2b57-40d9-a059-5532c25fe5e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-hvtgf\" (UID: \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:00.609205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.609156 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:00.672429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.672393 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/e77e5165-0098-48e8-8df6-ff4046a1d37b-kube-api-access-6sql4\") pod \"maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg\" (UID: \"e77e5165-0098-48e8-8df6-ff4046a1d37b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:00.673057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.672998 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmsw6\" (UniqueName: \"kubernetes.io/projected/9045c5cb-a562-4704-8bb9-5c63bf0c4585-kube-api-access-tmsw6\") pod \"maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x\" (UID: \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:00.673799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.673769 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4s8s\" (UniqueName: \"kubernetes.io/projected/e0db05bf-2b57-40d9-a059-5532c25fe5e3-kube-api-access-m4s8s\") pod \"maas-api-key-cleanup-29892885-hvtgf\" (UID: \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:00.701377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.698280 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:00.701377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.699717 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blm8c\" (UniqueName: \"kubernetes.io/projected/fdcadc29-3748-4c2b-80a2-60d946bca6e1-kube-api-access-blm8c\") pod \"maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh\" (UID: \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:00.701377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.699798 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcadc29-3748-4c2b-80a2-60d946bca6e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh\" (UID: \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:00.702559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.702497 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcadc29-3748-4c2b-80a2-60d946bca6e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh\" (UID: \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:00.739870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.739785 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blm8c\" (UniqueName: \"kubernetes.io/projected/fdcadc29-3748-4c2b-80a2-60d946bca6e1-kube-api-access-blm8c\") pod \"maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh\" (UID: \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:00.778989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.778675 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:00.829289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.817387 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:00.914772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.902942 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf"] Nov 01 22:45:00.935000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.934969 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb"] Nov 01 22:45:00.965940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.965911 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" event={"ID":"cdffff35-fc26-4169-8373-f92a0d756b26","Type":"ContainerStarted","Data":"aa789b20724889d7b1828c6d3ac1d84c66af2dc0bc7d06bc1892a6e131b5cda3"} Nov 01 22:45:00.968673 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:00.968651 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:00.988364 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:00.988308 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1427ec2e_5d9e_4f6a_b78f_a229cdd6bf93.slice/crio-da2ab4a152d679db80ed6ce1ee45a5bfeee8bd17820f6b9c30832d9c6ac6aaed WatchSource:0}: Error finding container da2ab4a152d679db80ed6ce1ee45a5bfeee8bd17820f6b9c30832d9c6ac6aaed: Status 404 returned error can't find the container with id da2ab4a152d679db80ed6ce1ee45a5bfeee8bd17820f6b9c30832d9c6ac6aaed Nov 01 22:45:01.214106 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.214072 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk"] Nov 01 22:45:01.224896 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:01.224864 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbdba91_5443_4ec3_9d7e_efe7e397d20a.slice/crio-b4d00241b28491d4631dfda61e9d1726c19677d6ce8fed114cb7cace41b9e0cd WatchSource:0}: Error finding container b4d00241b28491d4631dfda61e9d1726c19677d6ce8fed114cb7cace41b9e0cd: Status 404 returned error can't find the container with id b4d00241b28491d4631dfda61e9d1726c19677d6ce8fed114cb7cace41b9e0cd Nov 01 22:45:01.386937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.386757 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9"] Nov 01 22:45:01.388940 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:01.388908 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7ab1ac_2a1c_40cf_b39b_19981d199020.slice/crio-5b46392128f8d6124783ead976f148d44c679e24cda72e3c96298efa2b819b10 WatchSource:0}: Error finding container 5b46392128f8d6124783ead976f148d44c679e24cda72e3c96298efa2b819b10: Status 404 returned error can't find the container with id 5b46392128f8d6124783ead976f148d44c679e24cda72e3c96298efa2b819b10 Nov 01 22:45:01.398316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.398296 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:45:01.416717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.416692 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5"] Nov 01 22:45:01.418471 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:01.418441 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c8b3b3_4de9_430a_a5df_ad824446bead.slice/crio-e05def52769ba41acba33e91f4f88c2e63a1d95854fe01d70e2268f4b4c3bbc9 WatchSource:0}: Error finding container e05def52769ba41acba33e91f4f88c2e63a1d95854fe01d70e2268f4b4c3bbc9: Status 404 returned error can't find the container with id e05def52769ba41acba33e91f4f88c2e63a1d95854fe01d70e2268f4b4c3bbc9 Nov 01 22:45:01.435743 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:01.435701 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8e2476_fb3a_496a_9f5a_ba753a927d97.slice/crio-c55df7e356cd5710f5506ac80c4f0260378e21745154a7617181c0919d403bb6 WatchSource:0}: Error finding container c55df7e356cd5710f5506ac80c4f0260378e21745154a7617181c0919d403bb6: Status 404 returned error can't find the container with id c55df7e356cd5710f5506ac80c4f0260378e21745154a7617181c0919d403bb6 Nov 01 22:45:01.437907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.437882 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm"] Nov 01 22:45:01.484732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.484698 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-xks-ca\") pod \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " Nov 01 22:45:01.484878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.484804 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2dfd10d-dedb-48e2-a725-1f3efd485f46-kube-api-access-wtbj6\" (UniqueName: \"kubernetes.io/projected/d2dfd10d-dedb-48e2-a725-1f3efd485f46-kube-api-access-wtbj6\") pod \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " Nov 01 22:45:01.484950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.484938 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-openshift-ca\") pod \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\" (UID: \"d2dfd10d-dedb-48e2-a725-1f3efd485f46\") " Nov 01 22:45:01.486184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.486100 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-openshift-ca" pod "d2dfd10d-dedb-48e2-a725-1f3efd485f46" (UID: "d2dfd10d-dedb-48e2-a725-1f3efd485f46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:01.574232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.574199 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2dfd10d-dedb-48e2-a725-1f3efd485f46-kube-api-access-wtbj6" pod "d2dfd10d-dedb-48e2-a725-1f3efd485f46" (UID: "d2dfd10d-dedb-48e2-a725-1f3efd485f46"). InnerVolumeSpecName "kube-api-access-wtbj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:01.574554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.574536 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-xks-ca" pod "d2dfd10d-dedb-48e2-a725-1f3efd485f46" (UID: "d2dfd10d-dedb-48e2-a725-1f3efd485f46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:01.588475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.588454 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:01.588602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.588581 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2dfd10d-dedb-48e2-a725-1f3efd485f46-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:01.588602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.588599 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbj6\" (UniqueName: \"kubernetes.io/projected/d2dfd10d-dedb-48e2-a725-1f3efd485f46-kube-api-access-wtbj6\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:01.612284 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.612250 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf"] Nov 01 22:45:01.617541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.617518 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr"] Nov 01 22:45:01.670315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.670286 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x"] Nov 01 22:45:01.676403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.676377 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg"] Nov 01 22:45:01.678345 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:01.678296 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6a9c68_a26b_4648_812b_1287c245cff0.slice/crio-7ae26c84ed00f04f8145549c8cd46444a6c3bc60041b534b485ca3239262bb41 WatchSource:0}: Error finding container 7ae26c84ed00f04f8145549c8cd46444a6c3bc60041b534b485ca3239262bb41: Status 404 returned error can't find the container with id 7ae26c84ed00f04f8145549c8cd46444a6c3bc60041b534b485ca3239262bb41 Nov 01 22:45:01.680140 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:01.680100 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0db05bf_2b57_40d9_a059_5532c25fe5e3.slice/crio-ca4354f0c23b2eb948fb5e956ee45d8ece45d704c30e80f12de9a45030c938d3 WatchSource:0}: Error finding container ca4354f0c23b2eb948fb5e956ee45d8ece45d704c30e80f12de9a45030c938d3: Status 404 returned error can't find the container with id ca4354f0c23b2eb948fb5e956ee45d8ece45d704c30e80f12de9a45030c938d3 Nov 01 22:45:01.681546 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:01.681518 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9045c5cb_a562_4704_8bb9_5c63bf0c4585.slice/crio-c2b7780b9cc969a10907a24584085446075331bb95ce1c2858600beabe7ad5e2 WatchSource:0}: Error finding container c2b7780b9cc969a10907a24584085446075331bb95ce1c2858600beabe7ad5e2: Status 404 returned error can't find the container with id c2b7780b9cc969a10907a24584085446075331bb95ce1c2858600beabe7ad5e2 Nov 01 22:45:01.682700 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:01.682678 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77e5165_0098_48e8_8df6_ff4046a1d37b.slice/crio-0cc9e8bc3882f073a8356884ae23950d988e8ecd159a6014e6b432021f716e59 WatchSource:0}: Error finding container 0cc9e8bc3882f073a8356884ae23950d988e8ecd159a6014e6b432021f716e59: Status 404 returned error can't find the container with id 0cc9e8bc3882f073a8356884ae23950d988e8ecd159a6014e6b432021f716e59 Nov 01 22:45:01.845403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.821755 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh"] Nov 01 22:45:01.978985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.978956 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" event={"ID":"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93","Type":"ContainerStarted","Data":"05e87132de452635fc1eb8f6a9e55214dc192ab0d7d33aeaefdcb94089251b62"} Nov 01 22:45:01.979104 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.979019 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" event={"ID":"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93","Type":"ContainerStarted","Data":"da2ab4a152d679db80ed6ce1ee45a5bfeee8bd17820f6b9c30832d9c6ac6aaed"} Nov 01 22:45:01.980502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.980470 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" event={"ID":"e77e5165-0098-48e8-8df6-ff4046a1d37b","Type":"ContainerStarted","Data":"0cc9e8bc3882f073a8356884ae23950d988e8ecd159a6014e6b432021f716e59"} Nov 01 22:45:01.981755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.981728 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" event={"ID":"dcbdba91-5443-4ec3-9d7e-efe7e397d20a","Type":"ContainerStarted","Data":"b4d00241b28491d4631dfda61e9d1726c19677d6ce8fed114cb7cace41b9e0cd"} Nov 01 22:45:01.983262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.983242 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" event={"ID":"d2dfd10d-dedb-48e2-a725-1f3efd485f46","Type":"ContainerDied","Data":"0b6f139d9916def25caaaefad80b01cc1883d19e6bf2331dd0e553b2cda575bd"} Nov 01 22:45:01.983262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.983257 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh" Nov 01 22:45:01.983405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.983276 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b6f139d9916def25caaaefad80b01cc1883d19e6bf2331dd0e553b2cda575bd" Nov 01 22:45:01.984635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.984615 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" event={"ID":"0b7ab1ac-2a1c-40cf-b39b-19981d199020","Type":"ContainerStarted","Data":"5b46392128f8d6124783ead976f148d44c679e24cda72e3c96298efa2b819b10"} Nov 01 22:45:01.986069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.986049 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" event={"ID":"eb8e2476-fb3a-496a-9f5a-ba753a927d97","Type":"ContainerStarted","Data":"c55df7e356cd5710f5506ac80c4f0260378e21745154a7617181c0919d403bb6"} Nov 01 22:45:01.987691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.987672 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" event={"ID":"a2c8b3b3-4de9-430a-a5df-ad824446bead","Type":"ContainerStarted","Data":"e05def52769ba41acba33e91f4f88c2e63a1d95854fe01d70e2268f4b4c3bbc9"} Nov 01 22:45:01.988986 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.988963 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" event={"ID":"fdcadc29-3748-4c2b-80a2-60d946bca6e1","Type":"ContainerStarted","Data":"e8a775e6d671ac16efbf3e87e9046212a8a43232ca2836a5ab33b2b6565fbfbc"} Nov 01 22:45:01.990234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.990211 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" event={"ID":"9045c5cb-a562-4704-8bb9-5c63bf0c4585","Type":"ContainerStarted","Data":"c2b7780b9cc969a10907a24584085446075331bb95ce1c2858600beabe7ad5e2"} Nov 01 22:45:01.991637 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.991610 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" event={"ID":"ca6a9c68-a26b-4648-812b-1287c245cff0","Type":"ContainerStarted","Data":"7ae26c84ed00f04f8145549c8cd46444a6c3bc60041b534b485ca3239262bb41"} Nov 01 22:45:01.992966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.992946 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" event={"ID":"e0db05bf-2b57-40d9-a059-5532c25fe5e3","Type":"ContainerStarted","Data":"ca4354f0c23b2eb948fb5e956ee45d8ece45d704c30e80f12de9a45030c938d3"} Nov 01 22:45:01.995289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.995265 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" event={"ID":"7cc7cc94-5af7-4618-8900-ce612c7a8ff8","Type":"ContainerStarted","Data":"95f350feaaef56019261409e8ee050085de2133989bd8a6ae7d4287c67cf8c37"} Nov 01 22:45:01.995379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.995306 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" event={"ID":"7cc7cc94-5af7-4618-8900-ce612c7a8ff8","Type":"ContainerStarted","Data":"68cf851ec43c04805265330e541c6ce897f080e35da83d62ca91ce140248fa88"} Nov 01 22:45:01.996989 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.996968 6287 generic.go:356] "Generic (PLEG): container finished" podID="cdffff35-fc26-4169-8373-f92a0d756b26" containerID="8f4155aca1c574d1d0df78abad1b649cae21491072c9b4c8c90af40b87d14771" exitCode=0 Nov 01 22:45:01.997083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:01.997001 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" event={"ID":"cdffff35-fc26-4169-8373-f92a0d756b26","Type":"ContainerDied","Data":"8f4155aca1c574d1d0df78abad1b649cae21491072c9b4c8c90af40b87d14771"} Nov 01 22:45:02.004254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:02.004213 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" podStartSLOduration=2.004202135 podStartE2EDuration="2.004202135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:01.999253161 +0000 UTC m=+1314.647752398" watchObservedRunningTime="2026-11-01 22:45:02.004202135 +0000 UTC m=+1314.652701369" Nov 01 22:45:02.050648 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:02.050592 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" podStartSLOduration=2.050575897 podStartE2EDuration="2.050575897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:02.043213353 +0000 UTC m=+1314.691712592" watchObservedRunningTime="2026-11-01 22:45:02.050575897 +0000 UTC m=+1314.699075133" Nov 01 22:45:03.007532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.007493 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" event={"ID":"e0db05bf-2b57-40d9-a059-5532c25fe5e3","Type":"ContainerStarted","Data":"7394fffc741e85bada043ee1064aa6f20453b6f655920a4b1c53eb06749ea9fb"} Nov 01 22:45:03.009376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.009349 6287 generic.go:356] "Generic (PLEG): container finished" podID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerID="95f350feaaef56019261409e8ee050085de2133989bd8a6ae7d4287c67cf8c37" exitCode=60 Nov 01 22:45:03.009512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.009434 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" event={"ID":"7cc7cc94-5af7-4618-8900-ce612c7a8ff8","Type":"ContainerDied","Data":"95f350feaaef56019261409e8ee050085de2133989bd8a6ae7d4287c67cf8c37"} Nov 01 22:45:03.009771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.009751 6287 scope.go:122] "RemoveContainer" containerID="95f350feaaef56019261409e8ee050085de2133989bd8a6ae7d4287c67cf8c37" Nov 01 22:45:03.011242 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.011216 6287 generic.go:356] "Generic (PLEG): container finished" podID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerID="05e87132de452635fc1eb8f6a9e55214dc192ab0d7d33aeaefdcb94089251b62" exitCode=60 Nov 01 22:45:03.011341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.011296 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" event={"ID":"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93","Type":"ContainerDied","Data":"05e87132de452635fc1eb8f6a9e55214dc192ab0d7d33aeaefdcb94089251b62"} Nov 01 22:45:03.011616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.011596 6287 scope.go:122] "RemoveContainer" containerID="05e87132de452635fc1eb8f6a9e55214dc192ab0d7d33aeaefdcb94089251b62" Nov 01 22:45:03.013537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.013511 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" event={"ID":"e77e5165-0098-48e8-8df6-ff4046a1d37b","Type":"ContainerStarted","Data":"b0521360309b80f6d59a9173d084c53e2c79650c8df63f0574162baaceca0b0e"} Nov 01 22:45:03.015553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.015533 6287 generic.go:356] "Generic (PLEG): container finished" podID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" containerID="9bcb0e06d63e05c9a218451ffde32ea6e4918703f5ac4eaf433a00e6891d1d14" exitCode=60 Nov 01 22:45:03.015653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.015620 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" event={"ID":"dcbdba91-5443-4ec3-9d7e-efe7e397d20a","Type":"ContainerDied","Data":"9bcb0e06d63e05c9a218451ffde32ea6e4918703f5ac4eaf433a00e6891d1d14"} Nov 01 22:45:03.016011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.015879 6287 scope.go:122] "RemoveContainer" containerID="9bcb0e06d63e05c9a218451ffde32ea6e4918703f5ac4eaf433a00e6891d1d14" Nov 01 22:45:03.017433 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.017415 6287 generic.go:356] "Generic (PLEG): container finished" podID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerID="16cbf682519f5367f282b00f3382e02a3ccc061945770ff94c1119b649fccb58" exitCode=60 Nov 01 22:45:03.017533 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.017497 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" event={"ID":"0b7ab1ac-2a1c-40cf-b39b-19981d199020","Type":"ContainerDied","Data":"16cbf682519f5367f282b00f3382e02a3ccc061945770ff94c1119b649fccb58"} Nov 01 22:45:03.017751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.017733 6287 scope.go:122] "RemoveContainer" containerID="16cbf682519f5367f282b00f3382e02a3ccc061945770ff94c1119b649fccb58" Nov 01 22:45:03.019397 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.019375 6287 generic.go:356] "Generic (PLEG): container finished" podID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerID="3c036502097a88ef5fdef13bf61b9c4ac123e77d5d9aaad41657e4402d35e9b4" exitCode=60 Nov 01 22:45:03.019570 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.019552 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" event={"ID":"eb8e2476-fb3a-496a-9f5a-ba753a927d97","Type":"ContainerDied","Data":"3c036502097a88ef5fdef13bf61b9c4ac123e77d5d9aaad41657e4402d35e9b4"} Nov 01 22:45:03.019672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.019653 6287 scope.go:122] "RemoveContainer" containerID="3c036502097a88ef5fdef13bf61b9c4ac123e77d5d9aaad41657e4402d35e9b4" Nov 01 22:45:03.021580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.021416 6287 generic.go:356] "Generic (PLEG): container finished" podID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerID="abe625dbc8da7c8dc8d59076c9e6e121d0ac449c065cc4499cde50419cf75cd0" exitCode=60 Nov 01 22:45:03.021580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.021521 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" event={"ID":"a2c8b3b3-4de9-430a-a5df-ad824446bead","Type":"ContainerDied","Data":"abe625dbc8da7c8dc8d59076c9e6e121d0ac449c065cc4499cde50419cf75cd0"} Nov 01 22:45:03.021887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.021869 6287 scope.go:122] "RemoveContainer" containerID="abe625dbc8da7c8dc8d59076c9e6e121d0ac449c065cc4499cde50419cf75cd0" Nov 01 22:45:03.023566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.023541 6287 generic.go:356] "Generic (PLEG): container finished" podID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerID="2b1a89438b80093321f2ada6598f64c21a19d96fdf43b1c0598086eabf069b4d" exitCode=60 Nov 01 22:45:03.023657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.023617 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" event={"ID":"fdcadc29-3748-4c2b-80a2-60d946bca6e1","Type":"ContainerDied","Data":"2b1a89438b80093321f2ada6598f64c21a19d96fdf43b1c0598086eabf069b4d"} Nov 01 22:45:03.023848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.023833 6287 scope.go:122] "RemoveContainer" containerID="2b1a89438b80093321f2ada6598f64c21a19d96fdf43b1c0598086eabf069b4d" Nov 01 22:45:03.027014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.026964 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" event={"ID":"9045c5cb-a562-4704-8bb9-5c63bf0c4585","Type":"ContainerStarted","Data":"93307d24c091c5e394d83ba95262338dd9922376e03d799c59e91c98894a711e"} Nov 01 22:45:03.028757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.028737 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" event={"ID":"ca6a9c68-a26b-4648-812b-1287c245cff0","Type":"ContainerStarted","Data":"f4ebbe74c03932f70f795cff1682ade33799d38834ec534d59c46ce00acbf231"} Nov 01 22:45:03.033695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.033647 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" podStartSLOduration=3.033631907 podStartE2EDuration="3.033631907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.02985384 +0000 UTC m=+1315.678353078" watchObservedRunningTime="2026-11-01 22:45:03.033631907 +0000 UTC m=+1315.682131138" Nov 01 22:45:03.111651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.111595 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" podStartSLOduration=3.111563448 podStartE2EDuration="3.111563448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.110082449 +0000 UTC m=+1315.758581678" watchObservedRunningTime="2026-11-01 22:45:03.111563448 +0000 UTC m=+1315.760062679" Nov 01 22:45:03.241186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.241129 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" podStartSLOduration=3.241092648 podStartE2EDuration="3.241092648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.230784276 +0000 UTC m=+1315.879283510" watchObservedRunningTime="2026-11-01 22:45:03.241092648 +0000 UTC m=+1315.889591884" Nov 01 22:45:03.267794 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.267739 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" podStartSLOduration=3.267720546 podStartE2EDuration="3.267720546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.265676918 +0000 UTC m=+1315.914176153" watchObservedRunningTime="2026-11-01 22:45:03.267720546 +0000 UTC m=+1315.916219784" Nov 01 22:45:03.474556 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.474529 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:45:03.570433 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.570340 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-xks-ca\") pod \"cdffff35-fc26-4169-8373-f92a0d756b26\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " Nov 01 22:45:03.571506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.570483 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdffff35-fc26-4169-8373-f92a0d756b26-kube-api-access-wx8rk\" (UniqueName: \"kubernetes.io/projected/cdffff35-fc26-4169-8373-f92a0d756b26-kube-api-access-wx8rk\") pod \"cdffff35-fc26-4169-8373-f92a0d756b26\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " Nov 01 22:45:03.571506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.570637 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-openshift-ca\") pod \"cdffff35-fc26-4169-8373-f92a0d756b26\" (UID: \"cdffff35-fc26-4169-8373-f92a0d756b26\") " Nov 01 22:45:03.571506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.571257 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-openshift-ca" pod "cdffff35-fc26-4169-8373-f92a0d756b26" (UID: "cdffff35-fc26-4169-8373-f92a0d756b26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:03.571965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.571940 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.574644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.574617 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdffff35-fc26-4169-8373-f92a0d756b26-kube-api-access-wx8rk" pod "cdffff35-fc26-4169-8373-f92a0d756b26" (UID: "cdffff35-fc26-4169-8373-f92a0d756b26"). InnerVolumeSpecName "kube-api-access-wx8rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:03.574737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.574614 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-xks-ca" pod "cdffff35-fc26-4169-8373-f92a0d756b26" (UID: "cdffff35-fc26-4169-8373-f92a0d756b26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:03.679680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.679644 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx8rk\" (UniqueName: \"kubernetes.io/projected/cdffff35-fc26-4169-8373-f92a0d756b26-kube-api-access-wx8rk\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:03.679845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:03.679696 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdffff35-fc26-4169-8373-f92a0d756b26-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:04.039863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.039813 6287 generic.go:356] "Generic (PLEG): container finished" podID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerID="93307d24c091c5e394d83ba95262338dd9922376e03d799c59e91c98894a711e" exitCode=60 Nov 01 22:45:04.040682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.039898 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" event={"ID":"9045c5cb-a562-4704-8bb9-5c63bf0c4585","Type":"ContainerDied","Data":"93307d24c091c5e394d83ba95262338dd9922376e03d799c59e91c98894a711e"} Nov 01 22:45:04.040682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.040240 6287 scope.go:122] "RemoveContainer" containerID="93307d24c091c5e394d83ba95262338dd9922376e03d799c59e91c98894a711e" Nov 01 22:45:04.041999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.041978 6287 generic.go:356] "Generic (PLEG): container finished" podID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerID="f4ebbe74c03932f70f795cff1682ade33799d38834ec534d59c46ce00acbf231" exitCode=60 Nov 01 22:45:04.042212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.042079 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" event={"ID":"ca6a9c68-a26b-4648-812b-1287c245cff0","Type":"ContainerDied","Data":"f4ebbe74c03932f70f795cff1682ade33799d38834ec534d59c46ce00acbf231"} Nov 01 22:45:04.042336 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.042311 6287 scope.go:122] "RemoveContainer" containerID="f4ebbe74c03932f70f795cff1682ade33799d38834ec534d59c46ce00acbf231" Nov 01 22:45:04.045575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.045548 6287 generic.go:356] "Generic (PLEG): container finished" podID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerID="7394fffc741e85bada043ee1064aa6f20453b6f655920a4b1c53eb06749ea9fb" exitCode=60 Nov 01 22:45:04.045666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.045644 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" event={"ID":"e0db05bf-2b57-40d9-a059-5532c25fe5e3","Type":"ContainerDied","Data":"7394fffc741e85bada043ee1064aa6f20453b6f655920a4b1c53eb06749ea9fb"} Nov 01 22:45:04.045838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.045823 6287 scope.go:122] "RemoveContainer" containerID="7394fffc741e85bada043ee1064aa6f20453b6f655920a4b1c53eb06749ea9fb" Nov 01 22:45:04.048032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.048005 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" event={"ID":"7cc7cc94-5af7-4618-8900-ce612c7a8ff8","Type":"ContainerStarted","Data":"f492057eeebe0cc73b75a37cb47d0a515d6298a76ea71a1884eee10295a97c42"} Nov 01 22:45:04.049477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.049460 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" Nov 01 22:45:04.049584 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.049474 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv" event={"ID":"cdffff35-fc26-4169-8373-f92a0d756b26","Type":"ContainerDied","Data":"aa789b20724889d7b1828c6d3ac1d84c66af2dc0bc7d06bc1892a6e131b5cda3"} Nov 01 22:45:04.049584 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.049510 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa789b20724889d7b1828c6d3ac1d84c66af2dc0bc7d06bc1892a6e131b5cda3" Nov 01 22:45:04.051721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.051703 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" event={"ID":"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93","Type":"ContainerStarted","Data":"db37e4fbc7f980bc580cf8741f770a272ee7c62562d85bc9856bcaf946318c5e"} Nov 01 22:45:04.053401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.053372 6287 generic.go:356] "Generic (PLEG): container finished" podID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerID="b0521360309b80f6d59a9173d084c53e2c79650c8df63f0574162baaceca0b0e" exitCode=60 Nov 01 22:45:04.053507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.053404 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" event={"ID":"e77e5165-0098-48e8-8df6-ff4046a1d37b","Type":"ContainerDied","Data":"b0521360309b80f6d59a9173d084c53e2c79650c8df63f0574162baaceca0b0e"} Nov 01 22:45:04.053680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.053660 6287 scope.go:122] "RemoveContainer" containerID="b0521360309b80f6d59a9173d084c53e2c79650c8df63f0574162baaceca0b0e" Nov 01 22:45:04.056756 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.056651 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" event={"ID":"dcbdba91-5443-4ec3-9d7e-efe7e397d20a","Type":"ContainerStarted","Data":"18d5a71144ef7ec93310e848b7144a0e7a97195e2c1e2225e8f13d820941322d"} Nov 01 22:45:04.092267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.091307 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" event={"ID":"0b7ab1ac-2a1c-40cf-b39b-19981d199020","Type":"ContainerStarted","Data":"5a224b27a51eec8586d5d90590476ef99b3decfe98fe2dca4a0906aa207e4d6d"} Nov 01 22:45:04.094780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.094720 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" event={"ID":"eb8e2476-fb3a-496a-9f5a-ba753a927d97","Type":"ContainerStarted","Data":"b8faa0b15b8adc2480123b4a1dabb6fdd2ed6f0c027ee894a0c73d50ea8a66ef"} Nov 01 22:45:04.097961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.097927 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" event={"ID":"a2c8b3b3-4de9-430a-a5df-ad824446bead","Type":"ContainerStarted","Data":"d03ff6ab6d0c9217455935c495b4d4e5eb76c8a44f44f32924f0448af2bf9999"} Nov 01 22:45:04.100190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.100161 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" event={"ID":"fdcadc29-3748-4c2b-80a2-60d946bca6e1","Type":"ContainerStarted","Data":"a41ec9365bcae06291f718bd3e0dede2880b07e00f074a7f3be71ba71f43a26e"} Nov 01 22:45:04.244389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.244342 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" podStartSLOduration=4.244327781 podStartE2EDuration="4.244327781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.232789778 +0000 UTC m=+1316.881289018" watchObservedRunningTime="2026-11-01 22:45:04.244327781 +0000 UTC m=+1316.892827015" Nov 01 22:45:04.257365 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.257038 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p"] Nov 01 22:45:04.263960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.263936 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7"] Nov 01 22:45:04.298201 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.298106 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs"] Nov 01 22:45:04.365571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.365515 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" podStartSLOduration=4.365501967 podStartE2EDuration="4.365501967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.364073281 +0000 UTC m=+1317.012572516" watchObservedRunningTime="2026-11-01 22:45:04.365501967 +0000 UTC m=+1317.014001203" Nov 01 22:45:04.405311 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.405265 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" podStartSLOduration=4.405249485 podStartE2EDuration="4.405249485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.401854239 +0000 UTC m=+1317.050353474" watchObservedRunningTime="2026-11-01 22:45:04.405249485 +0000 UTC m=+1317.053748730" Nov 01 22:45:04.469012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.468969 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" podStartSLOduration=4.468955382 podStartE2EDuration="4.468955382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.466056449 +0000 UTC m=+1317.114555683" watchObservedRunningTime="2026-11-01 22:45:04.468955382 +0000 UTC m=+1317.117454617" Nov 01 22:45:04.592173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:04.592101 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" podStartSLOduration=4.592086862 podStartE2EDuration="4.592086862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.590648131 +0000 UTC m=+1317.239147366" watchObservedRunningTime="2026-11-01 22:45:04.592086862 +0000 UTC m=+1317.240586097" Nov 01 22:45:05.135839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.135751 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" event={"ID":"e77e5165-0098-48e8-8df6-ff4046a1d37b","Type":"ContainerStarted","Data":"cc9d3ce67bad5b7fdce97100e00add651603d0957ee2d5e03e0a9a40d7966687"} Nov 01 22:45:05.142133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.142088 6287 generic.go:356] "Generic (PLEG): container finished" podID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" containerID="18d5a71144ef7ec93310e848b7144a0e7a97195e2c1e2225e8f13d820941322d" exitCode=60 Nov 01 22:45:05.142441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.142417 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" event={"ID":"dcbdba91-5443-4ec3-9d7e-efe7e397d20a","Type":"ContainerDied","Data":"18d5a71144ef7ec93310e848b7144a0e7a97195e2c1e2225e8f13d820941322d"} Nov 01 22:45:05.142614 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.142601 6287 scope.go:122] "RemoveContainer" containerID="9bcb0e06d63e05c9a218451ffde32ea6e4918703f5ac4eaf433a00e6891d1d14" Nov 01 22:45:05.151129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.143378 6287 scope.go:122] "RemoveContainer" containerID="18d5a71144ef7ec93310e848b7144a0e7a97195e2c1e2225e8f13d820941322d" Nov 01 22:45:05.151767 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:05.151740 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk_odh-ai-gateway-infra(dcbdba91-5443-4ec3-9d7e-efe7e397d20a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" podUID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" Nov 01 22:45:05.171294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.168146 6287 generic.go:356] "Generic (PLEG): container finished" podID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerID="5a224b27a51eec8586d5d90590476ef99b3decfe98fe2dca4a0906aa207e4d6d" exitCode=60 Nov 01 22:45:05.171294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.168289 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" event={"ID":"0b7ab1ac-2a1c-40cf-b39b-19981d199020","Type":"ContainerDied","Data":"5a224b27a51eec8586d5d90590476ef99b3decfe98fe2dca4a0906aa207e4d6d"} Nov 01 22:45:05.171294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.168920 6287 scope.go:122] "RemoveContainer" containerID="5a224b27a51eec8586d5d90590476ef99b3decfe98fe2dca4a0906aa207e4d6d" Nov 01 22:45:05.171294 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:05.169459 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9_odh-ai-gateway-infra(0b7ab1ac-2a1c-40cf-b39b-19981d199020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" Nov 01 22:45:05.184150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.183830 6287 generic.go:356] "Generic (PLEG): container finished" podID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerID="b8faa0b15b8adc2480123b4a1dabb6fdd2ed6f0c027ee894a0c73d50ea8a66ef" exitCode=60 Nov 01 22:45:05.184150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.184012 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" event={"ID":"eb8e2476-fb3a-496a-9f5a-ba753a927d97","Type":"ContainerDied","Data":"b8faa0b15b8adc2480123b4a1dabb6fdd2ed6f0c027ee894a0c73d50ea8a66ef"} Nov 01 22:45:05.184532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.184510 6287 scope.go:122] "RemoveContainer" containerID="b8faa0b15b8adc2480123b4a1dabb6fdd2ed6f0c027ee894a0c73d50ea8a66ef" Nov 01 22:45:05.185230 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:05.184966 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm_odh-ai-gateway-infra(eb8e2476-fb3a-496a-9f5a-ba753a927d97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" Nov 01 22:45:05.197546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.195413 6287 generic.go:356] "Generic (PLEG): container finished" podID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerID="d03ff6ab6d0c9217455935c495b4d4e5eb76c8a44f44f32924f0448af2bf9999" exitCode=60 Nov 01 22:45:05.197546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.195555 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" event={"ID":"a2c8b3b3-4de9-430a-a5df-ad824446bead","Type":"ContainerDied","Data":"d03ff6ab6d0c9217455935c495b4d4e5eb76c8a44f44f32924f0448af2bf9999"} Nov 01 22:45:05.197546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.196096 6287 scope.go:122] "RemoveContainer" containerID="d03ff6ab6d0c9217455935c495b4d4e5eb76c8a44f44f32924f0448af2bf9999" Nov 01 22:45:05.197546 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:05.196619 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5_odh-ai-gateway-infra(a2c8b3b3-4de9-430a-a5df-ad824446bead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" Nov 01 22:45:05.301935 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.301884 6287 scope.go:122] "RemoveContainer" containerID="16cbf682519f5367f282b00f3382e02a3ccc061945770ff94c1119b649fccb58" Nov 01 22:45:05.303847 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.303813 6287 generic.go:356] "Generic (PLEG): container finished" podID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerID="a41ec9365bcae06291f718bd3e0dede2880b07e00f074a7f3be71ba71f43a26e" exitCode=60 Nov 01 22:45:05.304552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.304247 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" event={"ID":"fdcadc29-3748-4c2b-80a2-60d946bca6e1","Type":"ContainerDied","Data":"a41ec9365bcae06291f718bd3e0dede2880b07e00f074a7f3be71ba71f43a26e"} Nov 01 22:45:05.306537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.306520 6287 scope.go:122] "RemoveContainer" containerID="a41ec9365bcae06291f718bd3e0dede2880b07e00f074a7f3be71ba71f43a26e" Nov 01 22:45:05.307322 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:05.307285 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh_odh-ai-gateway-infra(fdcadc29-3748-4c2b-80a2-60d946bca6e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" Nov 01 22:45:05.329144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.328973 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" event={"ID":"9045c5cb-a562-4704-8bb9-5c63bf0c4585","Type":"ContainerStarted","Data":"277a3e548d47c939f3151e8aaef243f3f1db5031ffb9daaac3b74201db6d4dfd"} Nov 01 22:45:05.337234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.337207 6287 scope.go:122] "RemoveContainer" containerID="3c036502097a88ef5fdef13bf61b9c4ac123e77d5d9aaad41657e4402d35e9b4" Nov 01 22:45:05.338391 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.338351 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" event={"ID":"ca6a9c68-a26b-4648-812b-1287c245cff0","Type":"ContainerStarted","Data":"0de337c1fa2a227ec7b635971dd941dc7831027c83ea995b7eeb6f9a2a8d3e8a"} Nov 01 22:45:05.343021 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.342992 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" event={"ID":"e0db05bf-2b57-40d9-a059-5532c25fe5e3","Type":"ContainerStarted","Data":"dad3796b7cf24c483e74a5ac97193a60111463e9f8815ac3a1ef9f6a8ac5fc09"} Nov 01 22:45:05.346273 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.346242 6287 generic.go:356] "Generic (PLEG): container finished" podID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerID="f492057eeebe0cc73b75a37cb47d0a515d6298a76ea71a1884eee10295a97c42" exitCode=60 Nov 01 22:45:05.346473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.346451 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" event={"ID":"7cc7cc94-5af7-4618-8900-ce612c7a8ff8","Type":"ContainerDied","Data":"f492057eeebe0cc73b75a37cb47d0a515d6298a76ea71a1884eee10295a97c42"} Nov 01 22:45:05.346564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.346540 6287 scope.go:122] "RemoveContainer" containerID="f492057eeebe0cc73b75a37cb47d0a515d6298a76ea71a1884eee10295a97c42" Nov 01 22:45:05.346840 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:05.346818 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf_odh-ai-gateway-infra(7cc7cc94-5af7-4618-8900-ce612c7a8ff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" Nov 01 22:45:05.349437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.349410 6287 generic.go:356] "Generic (PLEG): container finished" podID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerID="db37e4fbc7f980bc580cf8741f770a272ee7c62562d85bc9856bcaf946318c5e" exitCode=60 Nov 01 22:45:05.349545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.349448 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" event={"ID":"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93","Type":"ContainerDied","Data":"db37e4fbc7f980bc580cf8741f770a272ee7c62562d85bc9856bcaf946318c5e"} Nov 01 22:45:05.349678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.349663 6287 scope.go:122] "RemoveContainer" containerID="db37e4fbc7f980bc580cf8741f770a272ee7c62562d85bc9856bcaf946318c5e" Nov 01 22:45:05.349932 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:05.349914 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb_odh-ai-gateway-infra(1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" Nov 01 22:45:05.356373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.356346 6287 scope.go:122] "RemoveContainer" containerID="abe625dbc8da7c8dc8d59076c9e6e121d0ac449c065cc4499cde50419cf75cd0" Nov 01 22:45:05.376949 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.376246 6287 scope.go:122] "RemoveContainer" containerID="2b1a89438b80093321f2ada6598f64c21a19d96fdf43b1c0598086eabf069b4d" Nov 01 22:45:05.458350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.458323 6287 scope.go:122] "RemoveContainer" containerID="95f350feaaef56019261409e8ee050085de2133989bd8a6ae7d4287c67cf8c37" Nov 01 22:45:05.488518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:05.488492 6287 scope.go:122] "RemoveContainer" containerID="05e87132de452635fc1eb8f6a9e55214dc192ab0d7d33aeaefdcb94089251b62" Nov 01 22:45:06.360860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.360830 6287 scope.go:122] "RemoveContainer" containerID="db37e4fbc7f980bc580cf8741f770a272ee7c62562d85bc9856bcaf946318c5e" Nov 01 22:45:06.361296 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.361193 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb_odh-ai-gateway-infra(1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" Nov 01 22:45:06.362586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.362560 6287 generic.go:356] "Generic (PLEG): container finished" podID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerID="cc9d3ce67bad5b7fdce97100e00add651603d0957ee2d5e03e0a9a40d7966687" exitCode=60 Nov 01 22:45:06.362725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.362636 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" event={"ID":"e77e5165-0098-48e8-8df6-ff4046a1d37b","Type":"ContainerDied","Data":"cc9d3ce67bad5b7fdce97100e00add651603d0957ee2d5e03e0a9a40d7966687"} Nov 01 22:45:06.362725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.362693 6287 scope.go:122] "RemoveContainer" containerID="b0521360309b80f6d59a9173d084c53e2c79650c8df63f0574162baaceca0b0e" Nov 01 22:45:06.362879 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.362857 6287 scope.go:122] "RemoveContainer" containerID="cc9d3ce67bad5b7fdce97100e00add651603d0957ee2d5e03e0a9a40d7966687" Nov 01 22:45:06.363242 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.363219 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg_odh-ai-gateway-infra(e77e5165-0098-48e8-8df6-ff4046a1d37b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" Nov 01 22:45:06.364635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.364618 6287 scope.go:122] "RemoveContainer" containerID="18d5a71144ef7ec93310e848b7144a0e7a97195e2c1e2225e8f13d820941322d" Nov 01 22:45:06.364916 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.364893 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk_odh-ai-gateway-infra(dcbdba91-5443-4ec3-9d7e-efe7e397d20a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" podUID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" Nov 01 22:45:06.366307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.366287 6287 scope.go:122] "RemoveContainer" containerID="5a224b27a51eec8586d5d90590476ef99b3decfe98fe2dca4a0906aa207e4d6d" Nov 01 22:45:06.366592 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.366564 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9_odh-ai-gateway-infra(0b7ab1ac-2a1c-40cf-b39b-19981d199020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" Nov 01 22:45:06.367908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.367891 6287 scope.go:122] "RemoveContainer" containerID="b8faa0b15b8adc2480123b4a1dabb6fdd2ed6f0c027ee894a0c73d50ea8a66ef" Nov 01 22:45:06.368258 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.368238 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm_odh-ai-gateway-infra(eb8e2476-fb3a-496a-9f5a-ba753a927d97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" Nov 01 22:45:06.369650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.369632 6287 scope.go:122] "RemoveContainer" containerID="d03ff6ab6d0c9217455935c495b4d4e5eb76c8a44f44f32924f0448af2bf9999" Nov 01 22:45:06.369940 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.369914 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5_odh-ai-gateway-infra(a2c8b3b3-4de9-430a-a5df-ad824446bead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" Nov 01 22:45:06.371277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.371255 6287 scope.go:122] "RemoveContainer" containerID="a41ec9365bcae06291f718bd3e0dede2880b07e00f074a7f3be71ba71f43a26e" Nov 01 22:45:06.371535 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.371511 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh_odh-ai-gateway-infra(fdcadc29-3748-4c2b-80a2-60d946bca6e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" Nov 01 22:45:06.373031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.373006 6287 generic.go:356] "Generic (PLEG): container finished" podID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerID="277a3e548d47c939f3151e8aaef243f3f1db5031ffb9daaac3b74201db6d4dfd" exitCode=60 Nov 01 22:45:06.373201 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.373090 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" event={"ID":"9045c5cb-a562-4704-8bb9-5c63bf0c4585","Type":"ContainerDied","Data":"277a3e548d47c939f3151e8aaef243f3f1db5031ffb9daaac3b74201db6d4dfd"} Nov 01 22:45:06.373380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.373361 6287 scope.go:122] "RemoveContainer" containerID="277a3e548d47c939f3151e8aaef243f3f1db5031ffb9daaac3b74201db6d4dfd" Nov 01 22:45:06.373680 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.373657 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x_odh-ai-gateway-infra(9045c5cb-a562-4704-8bb9-5c63bf0c4585)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" Nov 01 22:45:06.382922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.382899 6287 generic.go:356] "Generic (PLEG): container finished" podID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerID="0de337c1fa2a227ec7b635971dd941dc7831027c83ea995b7eeb6f9a2a8d3e8a" exitCode=60 Nov 01 22:45:06.383014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.382981 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" event={"ID":"ca6a9c68-a26b-4648-812b-1287c245cff0","Type":"ContainerDied","Data":"0de337c1fa2a227ec7b635971dd941dc7831027c83ea995b7eeb6f9a2a8d3e8a"} Nov 01 22:45:06.383214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.383200 6287 scope.go:122] "RemoveContainer" containerID="0de337c1fa2a227ec7b635971dd941dc7831027c83ea995b7eeb6f9a2a8d3e8a" Nov 01 22:45:06.383590 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.383522 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr_odh-ai-gateway-infra(ca6a9c68-a26b-4648-812b-1287c245cff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" Nov 01 22:45:06.384785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.384770 6287 scope.go:122] "RemoveContainer" containerID="93307d24c091c5e394d83ba95262338dd9922376e03d799c59e91c98894a711e" Nov 01 22:45:06.385562 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.385545 6287 generic.go:356] "Generic (PLEG): container finished" podID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerID="dad3796b7cf24c483e74a5ac97193a60111463e9f8815ac3a1ef9f6a8ac5fc09" exitCode=60 Nov 01 22:45:06.385705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.385622 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" event={"ID":"e0db05bf-2b57-40d9-a059-5532c25fe5e3","Type":"ContainerDied","Data":"dad3796b7cf24c483e74a5ac97193a60111463e9f8815ac3a1ef9f6a8ac5fc09"} Nov 01 22:45:06.385811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.385796 6287 scope.go:122] "RemoveContainer" containerID="dad3796b7cf24c483e74a5ac97193a60111463e9f8815ac3a1ef9f6a8ac5fc09" Nov 01 22:45:06.386068 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.386047 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-hvtgf_odh-ai-gateway-infra(e0db05bf-2b57-40d9-a059-5532c25fe5e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" Nov 01 22:45:06.387669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.387644 6287 scope.go:122] "RemoveContainer" containerID="f492057eeebe0cc73b75a37cb47d0a515d6298a76ea71a1884eee10295a97c42" Nov 01 22:45:06.387895 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:06.387878 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf_odh-ai-gateway-infra(7cc7cc94-5af7-4618-8900-ce612c7a8ff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" Nov 01 22:45:06.401047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.401029 6287 scope.go:122] "RemoveContainer" containerID="f4ebbe74c03932f70f795cff1682ade33799d38834ec534d59c46ce00acbf231" Nov 01 22:45:06.416331 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:06.416309 6287 scope.go:122] "RemoveContainer" containerID="7394fffc741e85bada043ee1064aa6f20453b6f655920a4b1c53eb06749ea9fb" Nov 01 22:45:07.399567 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.399539 6287 scope.go:122] "RemoveContainer" containerID="277a3e548d47c939f3151e8aaef243f3f1db5031ffb9daaac3b74201db6d4dfd" Nov 01 22:45:07.400040 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:07.399908 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x_odh-ai-gateway-infra(9045c5cb-a562-4704-8bb9-5c63bf0c4585)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" Nov 01 22:45:07.401431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.401407 6287 scope.go:122] "RemoveContainer" containerID="0de337c1fa2a227ec7b635971dd941dc7831027c83ea995b7eeb6f9a2a8d3e8a" Nov 01 22:45:07.401710 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:07.401688 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr_odh-ai-gateway-infra(ca6a9c68-a26b-4648-812b-1287c245cff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" Nov 01 22:45:07.403172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.403147 6287 scope.go:122] "RemoveContainer" containerID="dad3796b7cf24c483e74a5ac97193a60111463e9f8815ac3a1ef9f6a8ac5fc09" Nov 01 22:45:07.403428 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:07.403405 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-hvtgf_odh-ai-gateway-infra(e0db05bf-2b57-40d9-a059-5532c25fe5e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" Nov 01 22:45:07.405352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.405332 6287 scope.go:122] "RemoveContainer" containerID="cc9d3ce67bad5b7fdce97100e00add651603d0957ee2d5e03e0a9a40d7966687" Nov 01 22:45:07.405605 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:07.405586 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg_odh-ai-gateway-infra(e77e5165-0098-48e8-8df6-ff4046a1d37b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" Nov 01 22:45:07.511029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.510993 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk"] Nov 01 22:45:07.696098 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.696063 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:07.818152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.816499 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-kube-api-access-dkg9k\" (UniqueName: \"kubernetes.io/projected/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-kube-api-access-dkg9k\") pod \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\" (UID: \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\") " Nov 01 22:45:07.818152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.816562 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-maas-api-service-ca\") pod \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\" (UID: \"dcbdba91-5443-4ec3-9d7e-efe7e397d20a\") " Nov 01 22:45:07.835149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.832941 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-maas-api-service-ca" pod "dcbdba91-5443-4ec3-9d7e-efe7e397d20a" (UID: "dcbdba91-5443-4ec3-9d7e-efe7e397d20a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:07.835149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.833271 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:07.919360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.912216 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-kube-api-access-dkg9k" pod "dcbdba91-5443-4ec3-9d7e-efe7e397d20a" (UID: "dcbdba91-5443-4ec3-9d7e-efe7e397d20a"). InnerVolumeSpecName "kube-api-access-dkg9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:08.043244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:07.939727 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkg9k\" (UniqueName: \"kubernetes.io/projected/dcbdba91-5443-4ec3-9d7e-efe7e397d20a-kube-api-access-dkg9k\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:08.452679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.452642 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz"] Nov 01 22:45:08.472169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.470489 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" event={"ID":"dcbdba91-5443-4ec3-9d7e-efe7e397d20a","Type":"ContainerDied","Data":"b4d00241b28491d4631dfda61e9d1726c19677d6ce8fed114cb7cace41b9e0cd"} Nov 01 22:45:08.472169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.470513 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk" Nov 01 22:45:08.472169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.470554 6287 scope.go:122] "RemoveContainer" containerID="18d5a71144ef7ec93310e848b7144a0e7a97195e2c1e2225e8f13d820941322d" Nov 01 22:45:08.476607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.476551 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b4f6bb-h68cz"] Nov 01 22:45:08.483077 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.483053 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv"] Nov 01 22:45:08.483289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.483267 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" containerID="cri-o://58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc" gracePeriod=30 Nov 01 22:45:08.623136 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.622668 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk"] Nov 01 22:45:08.657132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.650353 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b4f6bb-29892885-4d8zk"] Nov 01 22:45:08.739376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.739340 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542"] Nov 01 22:45:08.744029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:08.743999 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg"] Nov 01 22:45:09.476194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:09.476165 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58565c64-04fa-4507-b9c7-058493f53a38" path="/var/lib/kubelet/pods/58565c64-04fa-4507-b9c7-058493f53a38/volumes" Nov 01 22:45:09.476767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:09.476749 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" path="/var/lib/kubelet/pods/dcbdba91-5443-4ec3-9d7e-efe7e397d20a/volumes" Nov 01 22:45:09.944981 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:09.944945 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns"] Nov 01 22:45:09.966703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:09.966672 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw"] Nov 01 22:45:11.331541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:11.331502 6287 patch_prober.go:29] interesting pod/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:45:11.331964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:11.331562 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:45:13.743302 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:13.743141 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" podUID="3b07c554-3cc3-4163-a9fc-f0fec99f2992" containerName="payload-processing" containerID="cri-o://54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c" gracePeriod=25 Nov 01 22:45:13.744779 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:13.744728 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" podUID="0438e322-9ace-463d-a179-391ca2864d83" containerName="payload-pre-processing" containerID="cri-o://50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3" gracePeriod=25 Nov 01 22:45:13.926746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:13.926710 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:45:14.007032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.006982 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-socket\") pod \"479b82c5-9b9a-4abe-9372-38c5305bb753\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " Nov 01 22:45:14.007307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.007143 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-token\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-token\") pod \"479b82c5-9b9a-4abe-9372-38c5305bb753\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " Nov 01 22:45:14.007307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.007223 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479b82c5-9b9a-4abe-9372-38c5305bb753-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479b82c5-9b9a-4abe-9372-38c5305bb753-istiod-ca-cert\") pod \"479b82c5-9b9a-4abe-9372-38c5305bb753\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " Nov 01 22:45:14.007447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.007351 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-data\") pod \"479b82c5-9b9a-4abe-9372-38c5305bb753\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " Nov 01 22:45:14.007447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.007436 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-kube-api-access-vdq2g\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-kube-api-access-vdq2g\") pod \"479b82c5-9b9a-4abe-9372-38c5305bb753\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.009279 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-envoy\") pod \"479b82c5-9b9a-4abe-9372-38c5305bb753\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.009383 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-certs\") pod \"479b82c5-9b9a-4abe-9372-38c5305bb753\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.009422 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-podinfo\") pod \"479b82c5-9b9a-4abe-9372-38c5305bb753\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.009453 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-credential-socket\") pod \"479b82c5-9b9a-4abe-9372-38c5305bb753\" (UID: \"479b82c5-9b9a-4abe-9372-38c5305bb753\") " Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.010673 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-credential-socket" pod "479b82c5-9b9a-4abe-9372-38c5305bb753" (UID: "479b82c5-9b9a-4abe-9372-38c5305bb753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.011016 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479b82c5-9b9a-4abe-9372-38c5305bb753-istiod-ca-cert" pod "479b82c5-9b9a-4abe-9372-38c5305bb753" (UID: "479b82c5-9b9a-4abe-9372-38c5305bb753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.011139 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-socket" pod "479b82c5-9b9a-4abe-9372-38c5305bb753" (UID: "479b82c5-9b9a-4abe-9372-38c5305bb753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.011393 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-data" pod "479b82c5-9b9a-4abe-9372-38c5305bb753" (UID: "479b82c5-9b9a-4abe-9372-38c5305bb753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.011581 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-certs" pod "479b82c5-9b9a-4abe-9372-38c5305bb753" (UID: "479b82c5-9b9a-4abe-9372-38c5305bb753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.016608 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-envoy" pod "479b82c5-9b9a-4abe-9372-38c5305bb753" (UID: "479b82c5-9b9a-4abe-9372-38c5305bb753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:14.024382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.017861 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-token" pod "479b82c5-9b9a-4abe-9372-38c5305bb753" (UID: "479b82c5-9b9a-4abe-9372-38c5305bb753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.028354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.028314 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-podinfo" pod "479b82c5-9b9a-4abe-9372-38c5305bb753" (UID: "479b82c5-9b9a-4abe-9372-38c5305bb753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:14.028489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.028377 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-kube-api-access-vdq2g" pod "479b82c5-9b9a-4abe-9372-38c5305bb753" (UID: "479b82c5-9b9a-4abe-9372-38c5305bb753"). InnerVolumeSpecName "kube-api-access-vdq2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.115852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.115733 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.115852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.115787 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdq2g\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-kube-api-access-vdq2g\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.115852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.115804 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.115852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.115819 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.116370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.116313 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.116370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.116342 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.116594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.116521 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479b82c5-9b9a-4abe-9372-38c5305bb753-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.116594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.116542 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479b82c5-9b9a-4abe-9372-38c5305bb753-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.116594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.116557 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479b82c5-9b9a-4abe-9372-38c5305bb753-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.227323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.227026 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:45:14.232061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.232035 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:45:14.330158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.325682 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0438e322-9ace-463d-a179-391ca2864d83-kube-api-access-hchr5\" (UniqueName: \"kubernetes.io/projected/0438e322-9ace-463d-a179-391ca2864d83-kube-api-access-hchr5\") pod \"0438e322-9ace-463d-a179-391ca2864d83\" (UID: \"0438e322-9ace-463d-a179-391ca2864d83\") " Nov 01 22:45:14.330158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.326428 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b07c554-3cc3-4163-a9fc-f0fec99f2992-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b07c554-3cc3-4163-a9fc-f0fec99f2992-plugins-config-volume\") pod \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\" (UID: \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\") " Nov 01 22:45:14.330158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.326890 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b07c554-3cc3-4163-a9fc-f0fec99f2992-plugins-config-volume" pod "3b07c554-3cc3-4163-a9fc-f0fec99f2992" (UID: "3b07c554-3cc3-4163-a9fc-f0fec99f2992"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:14.330158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.326990 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b07c554-3cc3-4163-a9fc-f0fec99f2992-kube-api-access-dq6d5\" (UniqueName: \"kubernetes.io/projected/3b07c554-3cc3-4163-a9fc-f0fec99f2992-kube-api-access-dq6d5\") pod \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\" (UID: \"3b07c554-3cc3-4163-a9fc-f0fec99f2992\") " Nov 01 22:45:14.330158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.327377 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0438e322-9ace-463d-a179-391ca2864d83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438e322-9ace-463d-a179-391ca2864d83-plugins-config-volume\") pod \"0438e322-9ace-463d-a179-391ca2864d83\" (UID: \"0438e322-9ace-463d-a179-391ca2864d83\") " Nov 01 22:45:14.330158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.327794 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0438e322-9ace-463d-a179-391ca2864d83-plugins-config-volume" pod "0438e322-9ace-463d-a179-391ca2864d83" (UID: "0438e322-9ace-463d-a179-391ca2864d83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:14.330158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.328588 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b07c554-3cc3-4163-a9fc-f0fec99f2992-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.330158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.328612 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438e322-9ace-463d-a179-391ca2864d83-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.347640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.347579 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0438e322-9ace-463d-a179-391ca2864d83-kube-api-access-hchr5" pod "0438e322-9ace-463d-a179-391ca2864d83" (UID: "0438e322-9ace-463d-a179-391ca2864d83"). InnerVolumeSpecName "kube-api-access-hchr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.349487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.349446 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b07c554-3cc3-4163-a9fc-f0fec99f2992-kube-api-access-dq6d5" pod "3b07c554-3cc3-4163-a9fc-f0fec99f2992" (UID: "3b07c554-3cc3-4163-a9fc-f0fec99f2992"). InnerVolumeSpecName "kube-api-access-dq6d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:14.432533 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.432479 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6d5\" (UniqueName: \"kubernetes.io/projected/3b07c554-3cc3-4163-a9fc-f0fec99f2992-kube-api-access-dq6d5\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.432874 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.432841 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchr5\" (UniqueName: \"kubernetes.io/projected/0438e322-9ace-463d-a179-391ca2864d83-kube-api-access-hchr5\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:14.548378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.548338 6287 generic.go:356] "Generic (PLEG): container finished" podID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerID="58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc" exitCode=0 Nov 01 22:45:14.548620 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.548597 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" event={"ID":"479b82c5-9b9a-4abe-9372-38c5305bb753","Type":"ContainerDied","Data":"58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc"} Nov 01 22:45:14.548678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.548656 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" event={"ID":"479b82c5-9b9a-4abe-9372-38c5305bb753","Type":"ContainerDied","Data":"e8039b1eed9c600d7652b8c8b829489d23ec318afd15659df099646b02d720ce"} Nov 01 22:45:14.548724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.548686 6287 scope.go:122] "RemoveContainer" containerID="58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc" Nov 01 22:45:14.548878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.548862 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv" Nov 01 22:45:14.557540 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.557495 6287 generic.go:356] "Generic (PLEG): container finished" podID="0438e322-9ace-463d-a179-391ca2864d83" containerID="50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3" exitCode=0 Nov 01 22:45:14.558064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.557689 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" Nov 01 22:45:14.558064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.557777 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" event={"ID":"0438e322-9ace-463d-a179-391ca2864d83","Type":"ContainerDied","Data":"50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3"} Nov 01 22:45:14.558064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.557836 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg" event={"ID":"0438e322-9ace-463d-a179-391ca2864d83","Type":"ContainerDied","Data":"8a15cb92a985e57d2bb8bacdce3c3942cd4911324a0df1dbde23639ca6c1574e"} Nov 01 22:45:14.559781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.559759 6287 generic.go:356] "Generic (PLEG): container finished" podID="3b07c554-3cc3-4163-a9fc-f0fec99f2992" containerID="54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c" exitCode=0 Nov 01 22:45:14.559962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.559945 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" event={"ID":"3b07c554-3cc3-4163-a9fc-f0fec99f2992","Type":"ContainerDied","Data":"54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c"} Nov 01 22:45:14.560090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.560074 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" event={"ID":"3b07c554-3cc3-4163-a9fc-f0fec99f2992","Type":"ContainerDied","Data":"4fa6565de4714c1bf09999f0f0d5d2f51d9f06f87e3bd42a85df26cc2284eb4f"} Nov 01 22:45:14.560271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.560257 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542" Nov 01 22:45:14.602566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.602163 6287 scope.go:122] "RemoveContainer" containerID="56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585" Nov 01 22:45:14.611477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.609161 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv"] Nov 01 22:45:14.617319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.617296 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e25182-openshift-default-597bd67b47-tj2fv"] Nov 01 22:45:14.628498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.628473 6287 scope.go:122] "RemoveContainer" containerID="58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc" Nov 01 22:45:14.633335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.632164 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg"] Nov 01 22:45:14.634134 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:14.633913 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc\": container with ID starting with 58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc not found: ID does not exist" containerID="58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc" Nov 01 22:45:14.634134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.633955 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc"} err="failed to get container status \"58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc\": rpc error: code = NotFound desc = could not find container \"58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc\": container with ID starting with 58b0da06b374f07f2f534298dcd631703004e9fa583f35195e1890ae7310befc not found: ID does not exist" Nov 01 22:45:14.634134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.633982 6287 scope.go:122] "RemoveContainer" containerID="56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585" Nov 01 22:45:14.635648 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:14.635504 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585\": container with ID starting with 56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585 not found: ID does not exist" containerID="56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585" Nov 01 22:45:14.635648 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.635541 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585"} err="failed to get container status \"56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585\": rpc error: code = NotFound desc = could not find container \"56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585\": container with ID starting with 56332958dea81da0fe4339514fef2543f10901ad6e41fa9cbe68c1dc961e1585 not found: ID does not exist" Nov 01 22:45:14.635648 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.635562 6287 scope.go:122] "RemoveContainer" containerID="50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3" Nov 01 22:45:14.639221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.639201 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9c08e-58f8897767-7xpjg"] Nov 01 22:45:14.646891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.646869 6287 scope.go:122] "RemoveContainer" containerID="50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3" Nov 01 22:45:14.647182 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:14.647161 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3\": container with ID starting with 50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3 not found: ID does not exist" containerID="50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3" Nov 01 22:45:14.647241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.647190 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3"} err="failed to get container status \"50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3\": rpc error: code = NotFound desc = could not find container \"50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3\": container with ID starting with 50c20ce952aa999cf8bb1d1b589d6b6e717efdb53c536884a2243adc6e58a7d3 not found: ID does not exist" Nov 01 22:45:14.647241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.647210 6287 scope.go:122] "RemoveContainer" containerID="54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c" Nov 01 22:45:14.658204 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.658184 6287 scope.go:122] "RemoveContainer" containerID="54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c" Nov 01 22:45:14.658454 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:14.658435 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c\": container with ID starting with 54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c not found: ID does not exist" containerID="54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c" Nov 01 22:45:14.658506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.658460 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c"} err="failed to get container status \"54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c\": rpc error: code = NotFound desc = could not find container \"54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c\": container with ID starting with 54f52f42df5e25093a0b70d2b1222d7a5cd0598d4cebcffe991cd1ebfaef278c not found: ID does not exist" Nov 01 22:45:14.663058 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.663029 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542"] Nov 01 22:45:14.667951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.667930 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9c08e-5ffcc9685f-f9542"] Nov 01 22:45:14.945834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.945749 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" podUID="8ac2dd5a-11ff-408d-96f0-dac764508392" containerName="payload-processing" containerID="cri-o://6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785" gracePeriod=25 Nov 01 22:45:14.967038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:14.966976 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" podUID="9644b916-6603-48dc-b6e8-993bc7cec7f8" containerName="payload-pre-processing" containerID="cri-o://3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5" gracePeriod=25 Nov 01 22:45:15.464643 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:15.464610 6287 scope.go:122] "RemoveContainer" containerID="b8faa0b15b8adc2480123b4a1dabb6fdd2ed6f0c027ee894a0c73d50ea8a66ef" Nov 01 22:45:15.464852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:15.464727 6287 scope.go:122] "RemoveContainer" containerID="db37e4fbc7f980bc580cf8741f770a272ee7c62562d85bc9856bcaf946318c5e" Nov 01 22:45:15.464852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:15.464819 6287 scope.go:122] "RemoveContainer" containerID="5a224b27a51eec8586d5d90590476ef99b3decfe98fe2dca4a0906aa207e4d6d" Nov 01 22:45:15.509289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:15.504945 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0438e322-9ace-463d-a179-391ca2864d83" path="/var/lib/kubelet/pods/0438e322-9ace-463d-a179-391ca2864d83/volumes" Nov 01 22:45:15.509289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:15.505732 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b07c554-3cc3-4163-a9fc-f0fec99f2992" path="/var/lib/kubelet/pods/3b07c554-3cc3-4163-a9fc-f0fec99f2992/volumes" Nov 01 22:45:15.509289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:15.506522 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" path="/var/lib/kubelet/pods/479b82c5-9b9a-4abe-9372-38c5305bb753/volumes" Nov 01 22:45:15.621380 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:15.621347 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:45:16.464073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:16.463991 6287 scope.go:122] "RemoveContainer" containerID="dad3796b7cf24c483e74a5ac97193a60111463e9f8815ac3a1ef9f6a8ac5fc09" Nov 01 22:45:16.464461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:16.464180 6287 scope.go:122] "RemoveContainer" containerID="cc9d3ce67bad5b7fdce97100e00add651603d0957ee2d5e03e0a9a40d7966687" Nov 01 22:45:16.464461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:16.464275 6287 scope.go:122] "RemoveContainer" containerID="0de337c1fa2a227ec7b635971dd941dc7831027c83ea995b7eeb6f9a2a8d3e8a" Nov 01 22:45:16.464461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:16.464371 6287 scope.go:122] "RemoveContainer" containerID="f492057eeebe0cc73b75a37cb47d0a515d6298a76ea71a1884eee10295a97c42" Nov 01 22:45:16.464461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:16.464437 6287 scope.go:122] "RemoveContainer" containerID="d03ff6ab6d0c9217455935c495b4d4e5eb76c8a44f44f32924f0448af2bf9999" Nov 01 22:45:16.621465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:16.621136 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" event={"ID":"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93","Type":"ContainerStarted","Data":"bf524362d7a0f1e4bc8ccfc33e4d39488fc0def29a162e408a04422349bb1bc1"} Nov 01 22:45:16.626220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:16.623577 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" event={"ID":"0b7ab1ac-2a1c-40cf-b39b-19981d199020","Type":"ContainerStarted","Data":"5d97877a371b41ab62be8f4c1105e2ce7b133d06d6f89c9146b04a1901943142"} Nov 01 22:45:16.626511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:16.626435 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" event={"ID":"eb8e2476-fb3a-496a-9f5a-ba753a927d97","Type":"ContainerStarted","Data":"23e6cb61325d007270d7c3e901d9d05684ab6176a8ef5f8db4fa28c709f6ad4d"} Nov 01 22:45:17.641995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.641955 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" event={"ID":"e0db05bf-2b57-40d9-a059-5532c25fe5e3","Type":"ContainerStarted","Data":"9e453356ccb375a6991e71813b8685386c935ddb7c51573e9a54c9cc6b209867"} Nov 01 22:45:17.644575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.644544 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" event={"ID":"7cc7cc94-5af7-4618-8900-ce612c7a8ff8","Type":"ContainerStarted","Data":"e3394d26f17f8aff201eed34d4ff018a0140286ecb1832b3007325e615851b85"} Nov 01 22:45:17.650737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.649526 6287 generic.go:356] "Generic (PLEG): container finished" podID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerID="bf524362d7a0f1e4bc8ccfc33e4d39488fc0def29a162e408a04422349bb1bc1" exitCode=60 Nov 01 22:45:17.650737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.649616 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" event={"ID":"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93","Type":"ContainerDied","Data":"bf524362d7a0f1e4bc8ccfc33e4d39488fc0def29a162e408a04422349bb1bc1"} Nov 01 22:45:17.650737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.649663 6287 scope.go:122] "RemoveContainer" containerID="db37e4fbc7f980bc580cf8741f770a272ee7c62562d85bc9856bcaf946318c5e" Nov 01 22:45:17.650737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.650081 6287 scope.go:122] "RemoveContainer" containerID="bf524362d7a0f1e4bc8ccfc33e4d39488fc0def29a162e408a04422349bb1bc1" Nov 01 22:45:17.650737 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:17.650487 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb_odh-ai-gateway-infra(1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" Nov 01 22:45:17.655249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.655224 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" event={"ID":"e77e5165-0098-48e8-8df6-ff4046a1d37b","Type":"ContainerStarted","Data":"11ad2282d603ce232748b49f9e1f01707be8427223caf7267231efdde4ca5911"} Nov 01 22:45:17.657860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.657830 6287 generic.go:356] "Generic (PLEG): container finished" podID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerID="5d97877a371b41ab62be8f4c1105e2ce7b133d06d6f89c9146b04a1901943142" exitCode=60 Nov 01 22:45:17.657967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.657897 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" event={"ID":"0b7ab1ac-2a1c-40cf-b39b-19981d199020","Type":"ContainerDied","Data":"5d97877a371b41ab62be8f4c1105e2ce7b133d06d6f89c9146b04a1901943142"} Nov 01 22:45:17.658361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.658325 6287 scope.go:122] "RemoveContainer" containerID="5d97877a371b41ab62be8f4c1105e2ce7b133d06d6f89c9146b04a1901943142" Nov 01 22:45:17.658655 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:17.658629 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9_odh-ai-gateway-infra(0b7ab1ac-2a1c-40cf-b39b-19981d199020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" Nov 01 22:45:17.663609 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.663591 6287 generic.go:356] "Generic (PLEG): container finished" podID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerID="23e6cb61325d007270d7c3e901d9d05684ab6176a8ef5f8db4fa28c709f6ad4d" exitCode=60 Nov 01 22:45:17.663782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.663628 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" event={"ID":"eb8e2476-fb3a-496a-9f5a-ba753a927d97","Type":"ContainerDied","Data":"23e6cb61325d007270d7c3e901d9d05684ab6176a8ef5f8db4fa28c709f6ad4d"} Nov 01 22:45:17.664197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.664178 6287 scope.go:122] "RemoveContainer" containerID="23e6cb61325d007270d7c3e901d9d05684ab6176a8ef5f8db4fa28c709f6ad4d" Nov 01 22:45:17.664560 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:17.664538 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm_odh-ai-gateway-infra(eb8e2476-fb3a-496a-9f5a-ba753a927d97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" Nov 01 22:45:17.667355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.667332 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" event={"ID":"a2c8b3b3-4de9-430a-a5df-ad824446bead","Type":"ContainerStarted","Data":"ae36c31bae296b879205466eb9c9bab954a4c09396394a4618d95658e5084d75"} Nov 01 22:45:17.703657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.703458 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb"] Nov 01 22:45:17.713225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.713190 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" event={"ID":"ca6a9c68-a26b-4648-812b-1287c245cff0","Type":"ContainerStarted","Data":"c4a889a116954e59b92d7ea0d9d664123c08e06f35b7b3b7a88f85e247b399ba"} Nov 01 22:45:17.714148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.713765 6287 scope.go:122] "RemoveContainer" containerID="5a224b27a51eec8586d5d90590476ef99b3decfe98fe2dca4a0906aa207e4d6d" Nov 01 22:45:17.716258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.716185 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9"] Nov 01 22:45:17.740312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.739618 6287 scope.go:122] "RemoveContainer" containerID="b8faa0b15b8adc2480123b4a1dabb6fdd2ed6f0c027ee894a0c73d50ea8a66ef" Nov 01 22:45:17.753072 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:17.753035 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm"] Nov 01 22:45:18.464134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:18.463862 6287 scope.go:122] "RemoveContainer" containerID="a41ec9365bcae06291f718bd3e0dede2880b07e00f074a7f3be71ba71f43a26e" Nov 01 22:45:18.471134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:18.464915 6287 scope.go:122] "RemoveContainer" containerID="277a3e548d47c939f3151e8aaef243f3f1db5031ffb9daaac3b74201db6d4dfd" Nov 01 22:45:18.712459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:18.712362 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf"] Nov 01 22:45:19.018046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.017954 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg"] Nov 01 22:45:19.085838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.085803 6287 generic.go:356] "Generic (PLEG): container finished" podID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerID="11ad2282d603ce232748b49f9e1f01707be8427223caf7267231efdde4ca5911" exitCode=60 Nov 01 22:45:19.086172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.086146 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" event={"ID":"e77e5165-0098-48e8-8df6-ff4046a1d37b","Type":"ContainerDied","Data":"11ad2282d603ce232748b49f9e1f01707be8427223caf7267231efdde4ca5911"} Nov 01 22:45:19.086386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.086370 6287 scope.go:122] "RemoveContainer" containerID="cc9d3ce67bad5b7fdce97100e00add651603d0957ee2d5e03e0a9a40d7966687" Nov 01 22:45:19.211104 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.209288 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf"] Nov 01 22:45:19.216364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.216258 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr"] Nov 01 22:45:19.275170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.273476 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5"] Nov 01 22:45:19.326293 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.322692 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h"] Nov 01 22:45:19.326293 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.322949 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" containerID="cri-o://7ebada15c1c16daddb5b0258426b36981b407ff70243bb0a733048059580eb53" gracePeriod=30 Nov 01 22:45:19.329622 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.329592 6287 generic.go:356] "Generic (PLEG): container finished" podID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerID="ae36c31bae296b879205466eb9c9bab954a4c09396394a4618d95658e5084d75" exitCode=60 Nov 01 22:45:19.370739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.329985 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" event={"ID":"a2c8b3b3-4de9-430a-a5df-ad824446bead","Type":"ContainerDied","Data":"ae36c31bae296b879205466eb9c9bab954a4c09396394a4618d95658e5084d75"} Nov 01 22:45:19.382954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.382710 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" containerID="cri-o://7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7" gracePeriod=45 Nov 01 22:45:19.386188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.385665 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" containerID="cri-o://9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70" gracePeriod=45 Nov 01 22:45:19.388036 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.387586 6287 generic.go:356] "Generic (PLEG): container finished" podID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerID="c4a889a116954e59b92d7ea0d9d664123c08e06f35b7b3b7a88f85e247b399ba" exitCode=60 Nov 01 22:45:19.388036 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.387703 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" event={"ID":"ca6a9c68-a26b-4648-812b-1287c245cff0","Type":"ContainerDied","Data":"c4a889a116954e59b92d7ea0d9d664123c08e06f35b7b3b7a88f85e247b399ba"} Nov 01 22:45:19.401916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.399998 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl"] Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402353 6287 cpu_manager.go:414] "containerMap: removing container" podUID="cdffff35-fc26-4169-8373-f92a0d756b26" containerName="revoke-keys" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402375 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdffff35-fc26-4169-8373-f92a0d756b26" containerName="revoke-keys" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402402 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" containerName="cleanup" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402411 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" containerName="cleanup" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402471 6287 cpu_manager.go:414] "containerMap: removing container" podUID="d2dfd10d-dedb-48e2-a725-1f3efd485f46" containerName="revoke-keys" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402480 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2dfd10d-dedb-48e2-a725-1f3efd485f46" containerName="revoke-keys" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402498 6287 cpu_manager.go:414] "containerMap: removing container" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402506 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402548 6287 cpu_manager.go:414] "containerMap: removing container" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402555 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402566 6287 cpu_manager.go:414] "containerMap: removing container" podUID="3b07c554-3cc3-4163-a9fc-f0fec99f2992" containerName="payload-processing" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402573 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b07c554-3cc3-4163-a9fc-f0fec99f2992" containerName="payload-processing" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402585 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0438e322-9ace-463d-a179-391ca2864d83" containerName="payload-pre-processing" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402593 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0438e322-9ace-463d-a179-391ca2864d83" containerName="payload-pre-processing" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402767 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdffff35-fc26-4169-8373-f92a0d756b26" containerName="revoke-keys" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402782 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" containerName="cleanup" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402802 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2dfd10d-dedb-48e2-a725-1f3efd485f46" containerName="revoke-keys" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402815 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402826 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" containerName="cleanup" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402842 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="479b82c5-9b9a-4abe-9372-38c5305bb753" containerName="istio-proxy" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402852 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b07c554-3cc3-4163-a9fc-f0fec99f2992" containerName="payload-processing" Nov 01 22:45:19.403164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.402863 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0438e322-9ace-463d-a179-391ca2864d83" containerName="payload-pre-processing" Nov 01 22:45:19.405709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.404610 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" containerID="cri-o://1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88" gracePeriod=45 Nov 01 22:45:19.413431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.412523 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.418319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.417095 6287 generic.go:356] "Generic (PLEG): container finished" podID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerID="9e453356ccb375a6991e71813b8685386c935ddb7c51573e9a54c9cc6b209867" exitCode=60 Nov 01 22:45:19.418319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.417292 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" event={"ID":"e0db05bf-2b57-40d9-a059-5532c25fe5e3","Type":"ContainerDied","Data":"9e453356ccb375a6991e71813b8685386c935ddb7c51573e9a54c9cc6b209867"} Nov 01 22:45:19.425508 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.425437 6287 generic.go:356] "Generic (PLEG): container finished" podID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerID="e3394d26f17f8aff201eed34d4ff018a0140286ecb1832b3007325e615851b85" exitCode=60 Nov 01 22:45:19.428327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.426813 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" event={"ID":"7cc7cc94-5af7-4618-8900-ce612c7a8ff8","Type":"ContainerDied","Data":"e3394d26f17f8aff201eed34d4ff018a0140286ecb1832b3007325e615851b85"} Nov 01 22:45:19.451190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.440825 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:45:19.456696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.456162 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.456696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.456213 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbsq4\" (UniqueName: \"kubernetes.io/projected/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-kube-api-access-tbsq4\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.456696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.456388 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.505505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.505466 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl"] Nov 01 22:45:19.561613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.560985 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.561613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.561306 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.561613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.561345 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbsq4\" (UniqueName: \"kubernetes.io/projected/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-kube-api-access-tbsq4\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.568464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.568431 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.672240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.671613 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbsq4\" (UniqueName: \"kubernetes.io/projected/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-kube-api-access-tbsq4\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.672240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.671996 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.753577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.753446 6287 scope.go:122] "RemoveContainer" containerID="d03ff6ab6d0c9217455935c495b4d4e5eb76c8a44f44f32924f0448af2bf9999" Nov 01 22:45:19.794418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.794190 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:19.863414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.849960 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:19.890516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.889020 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-kube-api-access-rtlw4\" (UniqueName: \"kubernetes.io/projected/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-kube-api-access-rtlw4\") pod \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\" (UID: \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\") " Nov 01 22:45:19.890516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.889331 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-maas-api-service-ca\") pod \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\" (UID: \"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93\") " Nov 01 22:45:19.895871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.895840 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-kube-api-access-rtlw4" pod "1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" (UID: "1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93"). InnerVolumeSpecName "kube-api-access-rtlw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:19.896667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.896180 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-maas-api-service-ca" pod "1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" (UID: "1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:19.905338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.904953 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": dial tcp 10.133.0.97:8000: connect: connection refused" Nov 01 22:45:19.905338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.905249 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": dial tcp 10.133.0.62:8000: connect: connection refused" Nov 01 22:45:19.912372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.909007 6287 scope.go:122] "RemoveContainer" containerID="0de337c1fa2a227ec7b635971dd941dc7831027c83ea995b7eeb6f9a2a8d3e8a" Nov 01 22:45:19.934690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.934656 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:19.959456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.959415 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:19.964798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:19.964393 6287 scope.go:122] "RemoveContainer" containerID="dad3796b7cf24c483e74a5ac97193a60111463e9f8815ac3a1ef9f6a8ac5fc09" Nov 01 22:45:20.009139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.004208 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:20.016528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.016458 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.016528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.016489 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtlw4\" (UniqueName: \"kubernetes.io/projected/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93-kube-api-access-rtlw4\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.022472 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.018866 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" podUID="8ac2dd5a-11ff-408d-96f0-dac764508392" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:45:20.028052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.027713 6287 scope.go:122] "RemoveContainer" containerID="f492057eeebe0cc73b75a37cb47d0a515d6298a76ea71a1884eee10295a97c42" Nov 01 22:45:20.077983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.076807 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": dial tcp 10.133.0.59:8000: connect: connection refused" Nov 01 22:45:20.120154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.119265 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c8b3b3-4de9-430a-a5df-ad824446bead-kube-api-access-dxzfn\" (UniqueName: \"kubernetes.io/projected/a2c8b3b3-4de9-430a-a5df-ad824446bead-kube-api-access-dxzfn\") pod \"a2c8b3b3-4de9-430a-a5df-ad824446bead\" (UID: \"a2c8b3b3-4de9-430a-a5df-ad824446bead\") " Nov 01 22:45:20.120154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.119319 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7ab1ac-2a1c-40cf-b39b-19981d199020-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab1ac-2a1c-40cf-b39b-19981d199020-maas-api-service-ca\") pod \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\" (UID: \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\") " Nov 01 22:45:20.120154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.119368 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8e2476-fb3a-496a-9f5a-ba753a927d97-kube-api-access-mx5w4\" (UniqueName: \"kubernetes.io/projected/eb8e2476-fb3a-496a-9f5a-ba753a927d97-kube-api-access-mx5w4\") pod \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\" (UID: \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\") " Nov 01 22:45:20.120154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.119512 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7ab1ac-2a1c-40cf-b39b-19981d199020-kube-api-access-lh6cc\" (UniqueName: \"kubernetes.io/projected/0b7ab1ac-2a1c-40cf-b39b-19981d199020-kube-api-access-lh6cc\") pod \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\" (UID: \"0b7ab1ac-2a1c-40cf-b39b-19981d199020\") " Nov 01 22:45:20.120154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.119743 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c8b3b3-4de9-430a-a5df-ad824446bead-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c8b3b3-4de9-430a-a5df-ad824446bead-maas-api-service-ca\") pod \"a2c8b3b3-4de9-430a-a5df-ad824446bead\" (UID: \"a2c8b3b3-4de9-430a-a5df-ad824446bead\") " Nov 01 22:45:20.120154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.119847 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8e2476-fb3a-496a-9f5a-ba753a927d97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8e2476-fb3a-496a-9f5a-ba753a927d97-maas-api-service-ca\") pod \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\" (UID: \"eb8e2476-fb3a-496a-9f5a-ba753a927d97\") " Nov 01 22:45:20.122870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.121614 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c8b3b3-4de9-430a-a5df-ad824446bead-maas-api-service-ca" pod "a2c8b3b3-4de9-430a-a5df-ad824446bead" (UID: "a2c8b3b3-4de9-430a-a5df-ad824446bead"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.122870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.121944 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7ab1ac-2a1c-40cf-b39b-19981d199020-maas-api-service-ca" pod "0b7ab1ac-2a1c-40cf-b39b-19981d199020" (UID: "0b7ab1ac-2a1c-40cf-b39b-19981d199020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.129205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.124825 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8e2476-fb3a-496a-9f5a-ba753a927d97-maas-api-service-ca" pod "eb8e2476-fb3a-496a-9f5a-ba753a927d97" (UID: "eb8e2476-fb3a-496a-9f5a-ba753a927d97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.129205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.126294 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7ab1ac-2a1c-40cf-b39b-19981d199020-kube-api-access-lh6cc" pod "0b7ab1ac-2a1c-40cf-b39b-19981d199020" (UID: "0b7ab1ac-2a1c-40cf-b39b-19981d199020"). InnerVolumeSpecName "kube-api-access-lh6cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.129205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.127834 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c8b3b3-4de9-430a-a5df-ad824446bead-kube-api-access-dxzfn" pod "a2c8b3b3-4de9-430a-a5df-ad824446bead" (UID: "a2c8b3b3-4de9-430a-a5df-ad824446bead"). InnerVolumeSpecName "kube-api-access-dxzfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.133263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.130548 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8e2476-fb3a-496a-9f5a-ba753a927d97-kube-api-access-mx5w4" pod "eb8e2476-fb3a-496a-9f5a-ba753a927d97" (UID: "eb8e2476-fb3a-496a-9f5a-ba753a927d97"). InnerVolumeSpecName "kube-api-access-mx5w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.232050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.231934 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8e2476-fb3a-496a-9f5a-ba753a927d97-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.232050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.231968 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxzfn\" (UniqueName: \"kubernetes.io/projected/a2c8b3b3-4de9-430a-a5df-ad824446bead-kube-api-access-dxzfn\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.232050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.231983 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7ab1ac-2a1c-40cf-b39b-19981d199020-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.232050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.231996 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx5w4\" (UniqueName: \"kubernetes.io/projected/eb8e2476-fb3a-496a-9f5a-ba753a927d97-kube-api-access-mx5w4\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.232050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.232008 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6cc\" (UniqueName: \"kubernetes.io/projected/0b7ab1ac-2a1c-40cf-b39b-19981d199020-kube-api-access-lh6cc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.232050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.232024 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c8b3b3-4de9-430a-a5df-ad824446bead-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.264731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.258999 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:20.264731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.264136 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:20.305824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.305734 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:20.327557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.327237 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:20.334094 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.334013 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:45:20.337706 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.337332 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6a9c68-a26b-4648-812b-1287c245cff0-kube-api-access-hjcsn\" (UniqueName: \"kubernetes.io/projected/ca6a9c68-a26b-4648-812b-1287c245cff0-kube-api-access-hjcsn\") pod \"ca6a9c68-a26b-4648-812b-1287c245cff0\" (UID: \"ca6a9c68-a26b-4648-812b-1287c245cff0\") " Nov 01 22:45:20.338049 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.338028 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0db05bf-2b57-40d9-a059-5532c25fe5e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0db05bf-2b57-40d9-a059-5532c25fe5e3-maas-api-service-ca\") pod \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\" (UID: \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\") " Nov 01 22:45:20.338677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.338455 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:45:20.338858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.338581 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0db05bf-2b57-40d9-a059-5532c25fe5e3-maas-api-service-ca" pod "e0db05bf-2b57-40d9-a059-5532c25fe5e3" (UID: "e0db05bf-2b57-40d9-a059-5532c25fe5e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.339051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.339033 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca6a9c68-a26b-4648-812b-1287c245cff0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6a9c68-a26b-4648-812b-1287c245cff0-maas-api-service-ca\") pod \"ca6a9c68-a26b-4648-812b-1287c245cff0\" (UID: \"ca6a9c68-a26b-4648-812b-1287c245cff0\") " Nov 01 22:45:20.339586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.339488 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6a9c68-a26b-4648-812b-1287c245cff0-maas-api-service-ca" pod "ca6a9c68-a26b-4648-812b-1287c245cff0" (UID: "ca6a9c68-a26b-4648-812b-1287c245cff0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.341857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.341839 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77e5165-0098-48e8-8df6-ff4046a1d37b-kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/e77e5165-0098-48e8-8df6-ff4046a1d37b-kube-api-access-6sql4\") pod \"e77e5165-0098-48e8-8df6-ff4046a1d37b\" (UID: \"e77e5165-0098-48e8-8df6-ff4046a1d37b\") " Nov 01 22:45:20.354090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.348933 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:45:20.354090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.353366 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77e5165-0098-48e8-8df6-ff4046a1d37b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77e5165-0098-48e8-8df6-ff4046a1d37b-maas-api-service-ca\") pod \"e77e5165-0098-48e8-8df6-ff4046a1d37b\" (UID: \"e77e5165-0098-48e8-8df6-ff4046a1d37b\") " Nov 01 22:45:20.354090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.353398 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0db05bf-2b57-40d9-a059-5532c25fe5e3-kube-api-access-m4s8s\" (UniqueName: \"kubernetes.io/projected/e0db05bf-2b57-40d9-a059-5532c25fe5e3-kube-api-access-m4s8s\") pod \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\" (UID: \"e0db05bf-2b57-40d9-a059-5532c25fe5e3\") " Nov 01 22:45:20.354090 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.353819 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77e5165-0098-48e8-8df6-ff4046a1d37b-maas-api-service-ca" pod "e77e5165-0098-48e8-8df6-ff4046a1d37b" (UID: "e77e5165-0098-48e8-8df6-ff4046a1d37b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.354416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.354155 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0db05bf-2b57-40d9-a059-5532c25fe5e3-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.354416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.354176 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6a9c68-a26b-4648-812b-1287c245cff0-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.354416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.354191 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77e5165-0098-48e8-8df6-ff4046a1d37b-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.357451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.357281 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77e5165-0098-48e8-8df6-ff4046a1d37b-kube-api-access-6sql4" pod "e77e5165-0098-48e8-8df6-ff4046a1d37b" (UID: "e77e5165-0098-48e8-8df6-ff4046a1d37b"). InnerVolumeSpecName "kube-api-access-6sql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.357724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.357705 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6a9c68-a26b-4648-812b-1287c245cff0-kube-api-access-hjcsn" pod "ca6a9c68-a26b-4648-812b-1287c245cff0" (UID: "ca6a9c68-a26b-4648-812b-1287c245cff0"). InnerVolumeSpecName "kube-api-access-hjcsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.364687 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.361319 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0db05bf-2b57-40d9-a059-5532c25fe5e3-kube-api-access-m4s8s" pod "e0db05bf-2b57-40d9-a059-5532c25fe5e3" (UID: "e0db05bf-2b57-40d9-a059-5532c25fe5e3"). InnerVolumeSpecName "kube-api-access-m4s8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.464603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.464572 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/452829c2-d248-41ef-a116-6810384b6951-tls-certs\" (UniqueName: \"kubernetes.io/secret/452829c2-d248-41ef-a116-6810384b6951-tls-certs\") pod \"452829c2-d248-41ef-a116-6810384b6951\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " Nov 01 22:45:20.464801 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.464784 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-dshm\") pod \"42fbbcb1-cd8b-497b-9588-e3015413a83e\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " Nov 01 22:45:20.464968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.464952 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-tmp-dir\") pod \"29e2c051-67dd-41c7-8301-e5b5697c868e\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " Nov 01 22:45:20.465093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.465077 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-dshm\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-dshm\") pod \"452829c2-d248-41ef-a116-6810384b6951\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " Nov 01 22:45:20.465241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.465225 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-home\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-home\") pod \"452829c2-d248-41ef-a116-6810384b6951\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " Nov 01 22:45:20.465409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.465389 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fbbcb1-cd8b-497b-9588-e3015413a83e-tls-certs\" (UniqueName: \"kubernetes.io/secret/42fbbcb1-cd8b-497b-9588-e3015413a83e-tls-certs\") pod \"42fbbcb1-cd8b-497b-9588-e3015413a83e\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " Nov 01 22:45:20.465526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.465511 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-model-cache\") pod \"29e2c051-67dd-41c7-8301-e5b5697c868e\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " Nov 01 22:45:20.465679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.465662 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-maas-api-service-ca\") pod \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\" (UID: \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\") " Nov 01 22:45:20.465820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.465804 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-dshm\") pod \"29e2c051-67dd-41c7-8301-e5b5697c868e\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " Nov 01 22:45:20.465949 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.465932 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e2c051-67dd-41c7-8301-e5b5697c868e-tls-certs\" (UniqueName: \"kubernetes.io/secret/29e2c051-67dd-41c7-8301-e5b5697c868e-tls-certs\") pod \"29e2c051-67dd-41c7-8301-e5b5697c868e\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " Nov 01 22:45:20.466079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466065 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-tmp-dir\") pod \"42fbbcb1-cd8b-497b-9588-e3015413a83e\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " Nov 01 22:45:20.466249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466233 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-home\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-home\") pod \"29e2c051-67dd-41c7-8301-e5b5697c868e\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " Nov 01 22:45:20.467131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466380 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-model-cache\") pod \"452829c2-d248-41ef-a116-6810384b6951\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " Nov 01 22:45:20.467131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466463 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-model-cache\") pod \"42fbbcb1-cd8b-497b-9588-e3015413a83e\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " Nov 01 22:45:20.467131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466504 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e2c051-67dd-41c7-8301-e5b5697c868e-kube-api-access-pdbcd\" (UniqueName: \"kubernetes.io/projected/29e2c051-67dd-41c7-8301-e5b5697c868e-kube-api-access-pdbcd\") pod \"29e2c051-67dd-41c7-8301-e5b5697c868e\" (UID: \"29e2c051-67dd-41c7-8301-e5b5697c868e\") " Nov 01 22:45:20.467131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466614 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-tmp-dir" pod "29e2c051-67dd-41c7-8301-e5b5697c868e" (UID: "29e2c051-67dd-41c7-8301-e5b5697c868e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.467131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466666 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-kube-api-access-n5rld\" (UniqueName: \"kubernetes.io/projected/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-kube-api-access-n5rld\") pod \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\" (UID: \"7cc7cc94-5af7-4618-8900-ce612c7a8ff8\") " Nov 01 22:45:20.467131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466760 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fbbcb1-cd8b-497b-9588-e3015413a83e-kube-api-access-fvst6\" (UniqueName: \"kubernetes.io/projected/42fbbcb1-cd8b-497b-9588-e3015413a83e-kube-api-access-fvst6\") pod \"42fbbcb1-cd8b-497b-9588-e3015413a83e\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " Nov 01 22:45:20.467131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466793 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-tmp-dir\") pod \"452829c2-d248-41ef-a116-6810384b6951\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " Nov 01 22:45:20.467131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466830 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452829c2-d248-41ef-a116-6810384b6951-kube-api-access-njs89\" (UniqueName: \"kubernetes.io/projected/452829c2-d248-41ef-a116-6810384b6951-kube-api-access-njs89\") pod \"452829c2-d248-41ef-a116-6810384b6951\" (UID: \"452829c2-d248-41ef-a116-6810384b6951\") " Nov 01 22:45:20.467131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.466915 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-home\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-home\") pod \"42fbbcb1-cd8b-497b-9588-e3015413a83e\" (UID: \"42fbbcb1-cd8b-497b-9588-e3015413a83e\") " Nov 01 22:45:20.468371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.467956 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-model-cache" pod "29e2c051-67dd-41c7-8301-e5b5697c868e" (UID: "29e2c051-67dd-41c7-8301-e5b5697c868e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.468371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468092 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-tmp-dir" pod "42fbbcb1-cd8b-497b-9588-e3015413a83e" (UID: "42fbbcb1-cd8b-497b-9588-e3015413a83e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.468371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468227 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-home" pod "42fbbcb1-cd8b-497b-9588-e3015413a83e" (UID: "42fbbcb1-cd8b-497b-9588-e3015413a83e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.468606 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468456 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-home" pod "29e2c051-67dd-41c7-8301-e5b5697c868e" (UID: "29e2c051-67dd-41c7-8301-e5b5697c868e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.468916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468647 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.468916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468658 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-model-cache" pod "452829c2-d248-41ef-a116-6810384b6951" (UID: "452829c2-d248-41ef-a116-6810384b6951"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.468916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468673 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sql4\" (UniqueName: \"kubernetes.io/projected/e77e5165-0098-48e8-8df6-ff4046a1d37b-kube-api-access-6sql4\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.468916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468704 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.468916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468718 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.468916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468731 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.468916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468774 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4s8s\" (UniqueName: \"kubernetes.io/projected/e0db05bf-2b57-40d9-a059-5532c25fe5e3-kube-api-access-m4s8s\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.468916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468790 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjcsn\" (UniqueName: \"kubernetes.io/projected/ca6a9c68-a26b-4648-812b-1287c245cff0-kube-api-access-hjcsn\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.468916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468851 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-model-cache" pod "42fbbcb1-cd8b-497b-9588-e3015413a83e" (UID: "42fbbcb1-cd8b-497b-9588-e3015413a83e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.469513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.468804 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.477801 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.477774 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" event={"ID":"1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93","Type":"ContainerDied","Data":"da2ab4a152d679db80ed6ce1ee45a5bfeee8bd17820f6b9c30832d9c6ac6aaed"} Nov 01 22:45:20.477954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.477942 6287 scope.go:122] "RemoveContainer" containerID="bf524362d7a0f1e4bc8ccfc33e4d39488fc0def29a162e408a04422349bb1bc1" Nov 01 22:45:20.478218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.478205 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb" Nov 01 22:45:20.493363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.478249 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-tmp-dir" pod "452829c2-d248-41ef-a116-6810384b6951" (UID: "452829c2-d248-41ef-a116-6810384b6951"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.493363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.478611 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-maas-api-service-ca" pod "7cc7cc94-5af7-4618-8900-ce612c7a8ff8" (UID: "7cc7cc94-5af7-4618-8900-ce612c7a8ff8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.493363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.479580 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-home" pod "452829c2-d248-41ef-a116-6810384b6951" (UID: "452829c2-d248-41ef-a116-6810384b6951"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.493363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.490623 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452829c2-d248-41ef-a116-6810384b6951-kube-api-access-njs89" pod "452829c2-d248-41ef-a116-6810384b6951" (UID: "452829c2-d248-41ef-a116-6810384b6951"). InnerVolumeSpecName "kube-api-access-njs89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.493363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.491396 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-kube-api-access-n5rld" pod "7cc7cc94-5af7-4618-8900-ce612c7a8ff8" (UID: "7cc7cc94-5af7-4618-8900-ce612c7a8ff8"). InnerVolumeSpecName "kube-api-access-n5rld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.495475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.493864 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fbbcb1-cd8b-497b-9588-e3015413a83e-tls-certs" pod "42fbbcb1-cd8b-497b-9588-e3015413a83e" (UID: "42fbbcb1-cd8b-497b-9588-e3015413a83e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:20.495475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.494573 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" event={"ID":"e77e5165-0098-48e8-8df6-ff4046a1d37b","Type":"ContainerDied","Data":"0cc9e8bc3882f073a8356884ae23950d988e8ecd159a6014e6b432021f716e59"} Nov 01 22:45:20.495475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.494943 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-dshm" pod "452829c2-d248-41ef-a116-6810384b6951" (UID: "452829c2-d248-41ef-a116-6810384b6951"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.495475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.495070 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e2c051-67dd-41c7-8301-e5b5697c868e-tls-certs" pod "29e2c051-67dd-41c7-8301-e5b5697c868e" (UID: "29e2c051-67dd-41c7-8301-e5b5697c868e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:20.495475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.495157 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-dshm" pod "42fbbcb1-cd8b-497b-9588-e3015413a83e" (UID: "42fbbcb1-cd8b-497b-9588-e3015413a83e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.495475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.495190 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e2c051-67dd-41c7-8301-e5b5697c868e-kube-api-access-pdbcd" pod "29e2c051-67dd-41c7-8301-e5b5697c868e" (UID: "29e2c051-67dd-41c7-8301-e5b5697c868e"). InnerVolumeSpecName "kube-api-access-pdbcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.495475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.494734 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg" Nov 01 22:45:20.496660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.496635 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-dshm" pod "29e2c051-67dd-41c7-8301-e5b5697c868e" (UID: "29e2c051-67dd-41c7-8301-e5b5697c868e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:20.498493 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.498438 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" Nov 01 22:45:20.500192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.499661 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9" event={"ID":"0b7ab1ac-2a1c-40cf-b39b-19981d199020","Type":"ContainerDied","Data":"5b46392128f8d6124783ead976f148d44c679e24cda72e3c96298efa2b819b10"} Nov 01 22:45:20.504543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.504301 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" Nov 01 22:45:20.504543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.504494 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm" event={"ID":"eb8e2476-fb3a-496a-9f5a-ba753a927d97","Type":"ContainerDied","Data":"c55df7e356cd5710f5506ac80c4f0260378e21745154a7617181c0919d403bb6"} Nov 01 22:45:20.528187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.527857 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" Nov 01 22:45:20.528538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.528469 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" event={"ID":"29e2c051-67dd-41c7-8301-e5b5697c868e","Type":"ContainerDied","Data":"9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70"} Nov 01 22:45:20.528710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.528687 6287 generic.go:356] "Generic (PLEG): container finished" podID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerID="9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70" exitCode=0 Nov 01 22:45:20.528893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.528875 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p" event={"ID":"29e2c051-67dd-41c7-8301-e5b5697c868e","Type":"ContainerDied","Data":"e8c826689799775ac8f12df1fbff9e78d31ffeb5364a94068b2e579eabca4227"} Nov 01 22:45:20.531519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.530879 6287 generic.go:356] "Generic (PLEG): container finished" podID="452829c2-d248-41ef-a116-6810384b6951" containerID="1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88" exitCode=0 Nov 01 22:45:20.531519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.531007 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" event={"ID":"452829c2-d248-41ef-a116-6810384b6951","Type":"ContainerDied","Data":"1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88"} Nov 01 22:45:20.531519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.531035 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" event={"ID":"452829c2-d248-41ef-a116-6810384b6951","Type":"ContainerDied","Data":"8752769cf261a3139631897add97db4875acb94d5143ab150a1be9c3ca3a984a"} Nov 01 22:45:20.531519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.531094 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs" Nov 01 22:45:20.538407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.538383 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" event={"ID":"a2c8b3b3-4de9-430a-a5df-ad824446bead","Type":"ContainerDied","Data":"e05def52769ba41acba33e91f4f88c2e63a1d95854fe01d70e2268f4b4c3bbc9"} Nov 01 22:45:20.542724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.542688 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" event={"ID":"fdcadc29-3748-4c2b-80a2-60d946bca6e1","Type":"ContainerDied","Data":"6e2d60c8384364ef63dc56f078803b33c0ad130467f147e4bdea4c0987934c90"} Nov 01 22:45:20.543143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.542554 6287 generic.go:356] "Generic (PLEG): container finished" podID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerID="6e2d60c8384364ef63dc56f078803b33c0ad130467f147e4bdea4c0987934c90" exitCode=60 Nov 01 22:45:20.546133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.538581 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5" Nov 01 22:45:20.565168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.563511 6287 scope.go:122] "RemoveContainer" containerID="6e2d60c8384364ef63dc56f078803b33c0ad130467f147e4bdea4c0987934c90" Nov 01 22:45:20.565168 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:20.563960 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh_odh-ai-gateway-infra(fdcadc29-3748-4c2b-80a2-60d946bca6e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572301 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572327 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdbcd\" (UniqueName: \"kubernetes.io/projected/29e2c051-67dd-41c7-8301-e5b5697c868e-kube-api-access-pdbcd\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572343 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rld\" (UniqueName: \"kubernetes.io/projected/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-kube-api-access-n5rld\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572358 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572371 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njs89\" (UniqueName: \"kubernetes.io/projected/452829c2-d248-41ef-a116-6810384b6951-kube-api-access-njs89\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572384 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42fbbcb1-cd8b-497b-9588-e3015413a83e-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572396 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572409 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572421 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42fbbcb1-cd8b-497b-9588-e3015413a83e-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572433 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc7cc94-5af7-4618-8900-ce612c7a8ff8-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572446 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29e2c051-67dd-41c7-8301-e5b5697c868e-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572458 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e2c051-67dd-41c7-8301-e5b5697c868e-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.573582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.572470 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452829c2-d248-41ef-a116-6810384b6951-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.576157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.574911 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452829c2-d248-41ef-a116-6810384b6951-tls-certs" pod "452829c2-d248-41ef-a116-6810384b6951" (UID: "452829c2-d248-41ef-a116-6810384b6951"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:20.576157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.575038 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fbbcb1-cd8b-497b-9588-e3015413a83e-kube-api-access-fvst6" pod "42fbbcb1-cd8b-497b-9588-e3015413a83e" (UID: "42fbbcb1-cd8b-497b-9588-e3015413a83e"). InnerVolumeSpecName "kube-api-access-fvst6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.576157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.575171 6287 generic.go:356] "Generic (PLEG): container finished" podID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerID="22de36faaa5fef970efcae9bb1c8a6fc75f2e1d86bfc000e8e4f304d169a5b23" exitCode=60 Nov 01 22:45:20.576157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.575264 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" event={"ID":"9045c5cb-a562-4704-8bb9-5c63bf0c4585","Type":"ContainerDied","Data":"22de36faaa5fef970efcae9bb1c8a6fc75f2e1d86bfc000e8e4f304d169a5b23"} Nov 01 22:45:20.576157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.575659 6287 scope.go:122] "RemoveContainer" containerID="22de36faaa5fef970efcae9bb1c8a6fc75f2e1d86bfc000e8e4f304d169a5b23" Nov 01 22:45:20.576157 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:20.576016 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x_odh-ai-gateway-infra(9045c5cb-a562-4704-8bb9-5c63bf0c4585)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" Nov 01 22:45:20.581319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.579468 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" event={"ID":"ca6a9c68-a26b-4648-812b-1287c245cff0","Type":"ContainerDied","Data":"7ae26c84ed00f04f8145549c8cd46444a6c3bc60041b534b485ca3239262bb41"} Nov 01 22:45:20.581319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.579541 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr" Nov 01 22:45:20.583760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.583726 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" Nov 01 22:45:20.589297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.584189 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf" event={"ID":"e0db05bf-2b57-40d9-a059-5532c25fe5e3","Type":"ContainerDied","Data":"ca4354f0c23b2eb948fb5e956ee45d8ece45d704c30e80f12de9a45030c938d3"} Nov 01 22:45:20.589297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.586003 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" event={"ID":"7cc7cc94-5af7-4618-8900-ce612c7a8ff8","Type":"ContainerDied","Data":"68cf851ec43c04805265330e541c6ce897f080e35da83d62ca91ce140248fa88"} Nov 01 22:45:20.589297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.586085 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf" Nov 01 22:45:20.600572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.600404 6287 scope.go:122] "RemoveContainer" containerID="11ad2282d603ce232748b49f9e1f01707be8427223caf7267231efdde4ca5911" Nov 01 22:45:20.605714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.604547 6287 generic.go:356] "Generic (PLEG): container finished" podID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerID="7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7" exitCode=0 Nov 01 22:45:20.605714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.604607 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" event={"ID":"42fbbcb1-cd8b-497b-9588-e3015413a83e","Type":"ContainerDied","Data":"7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7"} Nov 01 22:45:20.605714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.604643 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" event={"ID":"42fbbcb1-cd8b-497b-9588-e3015413a83e","Type":"ContainerDied","Data":"f24abf491786727f1481bbda6792556b5937ddabc7b30f0045640837d13d7ba5"} Nov 01 22:45:20.605714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.604715 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7" Nov 01 22:45:20.605714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.605661 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl"] Nov 01 22:45:20.641944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.641921 6287 scope.go:122] "RemoveContainer" containerID="5d97877a371b41ab62be8f4c1105e2ce7b133d06d6f89c9146b04a1901943142" Nov 01 22:45:20.679626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.679592 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvst6\" (UniqueName: \"kubernetes.io/projected/42fbbcb1-cd8b-497b-9588-e3015413a83e-kube-api-access-fvst6\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.679626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.679628 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452829c2-d248-41ef-a116-6810384b6951-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.685022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.685001 6287 scope.go:122] "RemoveContainer" containerID="23e6cb61325d007270d7c3e901d9d05684ab6176a8ef5f8db4fa28c709f6ad4d" Nov 01 22:45:20.696312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.696289 6287 scope.go:122] "RemoveContainer" containerID="9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70" Nov 01 22:45:20.707760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.707739 6287 scope.go:122] "RemoveContainer" containerID="9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70" Nov 01 22:45:20.708066 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:20.708042 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70\": container with ID starting with 9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70 not found: ID does not exist" containerID="9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70" Nov 01 22:45:20.708243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.708076 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70"} err="failed to get container status \"9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70\": rpc error: code = NotFound desc = could not find container \"9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70\": container with ID starting with 9a81ddf1c88be9f530c3c5d1c1f396f7353c9aeac25849ff95500abe69ff8a70 not found: ID does not exist" Nov 01 22:45:20.708243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.708100 6287 scope.go:122] "RemoveContainer" containerID="1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88" Nov 01 22:45:20.721238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.721214 6287 scope.go:122] "RemoveContainer" containerID="1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88" Nov 01 22:45:20.721628 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:20.721601 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88\": container with ID starting with 1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88 not found: ID does not exist" containerID="1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88" Nov 01 22:45:20.721752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.721637 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88"} err="failed to get container status \"1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88\": rpc error: code = NotFound desc = could not find container \"1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88\": container with ID starting with 1c1499d8e50fe6229e6d9917ace8948d9ac5c2a9ccc89689517b094b08af0d88 not found: ID does not exist" Nov 01 22:45:20.721752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.721659 6287 scope.go:122] "RemoveContainer" containerID="ae36c31bae296b879205466eb9c9bab954a4c09396394a4618d95658e5084d75" Nov 01 22:45:20.729356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.729332 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr"] Nov 01 22:45:20.747278 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.747127 6287 scope.go:122] "RemoveContainer" containerID="a41ec9365bcae06291f718bd3e0dede2880b07e00f074a7f3be71ba71f43a26e" Nov 01 22:45:20.762276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.758485 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-41b8f5-29892885-tv9zr"] Nov 01 22:45:20.779753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.779727 6287 scope.go:122] "RemoveContainer" containerID="277a3e548d47c939f3151e8aaef243f3f1db5031ffb9daaac3b74201db6d4dfd" Nov 01 22:45:20.779898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.779874 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm"] Nov 01 22:45:20.786344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.786323 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-381659-29892885-8jkwm"] Nov 01 22:45:20.797305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.797281 6287 scope.go:122] "RemoveContainer" containerID="c4a889a116954e59b92d7ea0d9d664123c08e06f35b7b3b7a88f85e247b399ba" Nov 01 22:45:20.805930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.805907 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf"] Nov 01 22:45:20.815151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.815102 6287 scope.go:122] "RemoveContainer" containerID="9e453356ccb375a6991e71813b8685386c935ddb7c51573e9a54c9cc6b209867" Nov 01 22:45:20.821684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.821659 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-00e49d-29892885-bcfmf"] Nov 01 22:45:20.825966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.825916 6287 scope.go:122] "RemoveContainer" containerID="e3394d26f17f8aff201eed34d4ff018a0140286ecb1832b3007325e615851b85" Nov 01 22:45:20.840315 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.840268 6287 scope.go:122] "RemoveContainer" containerID="7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7" Nov 01 22:45:20.847335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.847285 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p"] Nov 01 22:45:20.853012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.852984 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/distinct-w4-b42cc4-kserve-558ccb4959-s6k2p"] Nov 01 22:45:20.862740 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.862718 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg"] Nov 01 22:45:20.864797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.864756 6287 scope.go:122] "RemoveContainer" containerID="7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7" Nov 01 22:45:20.865223 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:20.865187 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7\": container with ID starting with 7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7 not found: ID does not exist" containerID="7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7" Nov 01 22:45:20.865223 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.865231 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7"} err="failed to get container status \"7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7\": rpc error: code = NotFound desc = could not find container \"7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7\": container with ID starting with 7e957ec7c87b289b37da56958fcdde3c4058ce2408d0ad9f382a0123accad1b7 not found: ID does not exist" Nov 01 22:45:20.869548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.869517 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b42cc4-29892885-f5qlg"] Nov 01 22:45:20.886999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.886974 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb"] Nov 01 22:45:20.891182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.891155 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dcd5d4-29892885-s8hxb"] Nov 01 22:45:20.912343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.912316 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9"] Nov 01 22:45:20.919971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.919942 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fa8cda-29892885-s4hm9"] Nov 01 22:45:20.927233 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.927209 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf"] Nov 01 22:45:20.936066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.936040 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-hvtgf"] Nov 01 22:45:20.952950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.952923 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5"] Nov 01 22:45:20.960353 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:20.960292 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-c9a7ce-29892885-q4br5"] Nov 01 22:45:21.000829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.000802 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7"] Nov 01 22:45:21.010298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.010272 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/premium-w4-b42cc4-kserve-6fdc975fbd-c7kl7"] Nov 01 22:45:21.027411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.027384 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs"] Nov 01 22:45:21.033149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.033127 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b42cc4/simulator-w4-b42cc4-kserve-78654f6f5c-hrqrs"] Nov 01 22:45:21.043868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.043837 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" podUID="9644b916-6603-48dc-b6e8-993bc7cec7f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:45:21.476207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.476174 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" path="/var/lib/kubelet/pods/0b7ab1ac-2a1c-40cf-b39b-19981d199020/volumes" Nov 01 22:45:21.476668 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.476649 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" path="/var/lib/kubelet/pods/1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93/volumes" Nov 01 22:45:21.477083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.477070 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" path="/var/lib/kubelet/pods/29e2c051-67dd-41c7-8301-e5b5697c868e/volumes" Nov 01 22:45:21.478369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.478352 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" path="/var/lib/kubelet/pods/42fbbcb1-cd8b-497b-9588-e3015413a83e/volumes" Nov 01 22:45:21.478773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.478761 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452829c2-d248-41ef-a116-6810384b6951" path="/var/lib/kubelet/pods/452829c2-d248-41ef-a116-6810384b6951/volumes" Nov 01 22:45:21.479189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.479174 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" path="/var/lib/kubelet/pods/7cc7cc94-5af7-4618-8900-ce612c7a8ff8/volumes" Nov 01 22:45:21.479865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.479850 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" path="/var/lib/kubelet/pods/a2c8b3b3-4de9-430a-a5df-ad824446bead/volumes" Nov 01 22:45:21.480265 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.480252 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" path="/var/lib/kubelet/pods/ca6a9c68-a26b-4648-812b-1287c245cff0/volumes" Nov 01 22:45:21.480633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.480621 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" path="/var/lib/kubelet/pods/e0db05bf-2b57-40d9-a059-5532c25fe5e3/volumes" Nov 01 22:45:21.481326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.481311 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" path="/var/lib/kubelet/pods/e77e5165-0098-48e8-8df6-ff4046a1d37b/volumes" Nov 01 22:45:21.481700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.481688 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" path="/var/lib/kubelet/pods/eb8e2476-fb3a-496a-9f5a-ba753a927d97/volumes" Nov 01 22:45:21.630067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.630030 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" event={"ID":"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3","Type":"ContainerStarted","Data":"2883be8cced56a999ccd07f760edfc8aacf84eeadd06007478b4384d4c6b0103"} Nov 01 22:45:21.630258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.630086 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" event={"ID":"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3","Type":"ContainerStarted","Data":"41adcd7d00befa52e5d4432ceb4a38cf8169371a274b958c88d604254a12dedb"} Nov 01 22:45:21.660404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.660351 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" podStartSLOduration=3.660336721 podStartE2EDuration="3.660336721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:21.656427108 +0000 UTC m=+1334.304926345" watchObservedRunningTime="2026-11-01 22:45:21.660336721 +0000 UTC m=+1334.308835955" Nov 01 22:45:21.683861 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.683822 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh"] Nov 01 22:45:21.887423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.887388 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:21.994936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.994899 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x"] Nov 01 22:45:22.000061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:21.999989 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdcadc29-3748-4c2b-80a2-60d946bca6e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcadc29-3748-4c2b-80a2-60d946bca6e1-maas-api-service-ca\") pod \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\" (UID: \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\") " Nov 01 22:45:22.000308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.000293 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcadc29-3748-4c2b-80a2-60d946bca6e1-kube-api-access-blm8c\" (UniqueName: \"kubernetes.io/projected/fdcadc29-3748-4c2b-80a2-60d946bca6e1-kube-api-access-blm8c\") pod \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\" (UID: \"fdcadc29-3748-4c2b-80a2-60d946bca6e1\") " Nov 01 22:45:22.000473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.000444 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcadc29-3748-4c2b-80a2-60d946bca6e1-maas-api-service-ca" pod "fdcadc29-3748-4c2b-80a2-60d946bca6e1" (UID: "fdcadc29-3748-4c2b-80a2-60d946bca6e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:22.000913 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.000876 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcadc29-3748-4c2b-80a2-60d946bca6e1-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:22.002628 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.002603 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcadc29-3748-4c2b-80a2-60d946bca6e1-kube-api-access-blm8c" pod "fdcadc29-3748-4c2b-80a2-60d946bca6e1" (UID: "fdcadc29-3748-4c2b-80a2-60d946bca6e1"). InnerVolumeSpecName "kube-api-access-blm8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:22.105744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.105704 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blm8c\" (UniqueName: \"kubernetes.io/projected/fdcadc29-3748-4c2b-80a2-60d946bca6e1-kube-api-access-blm8c\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:22.177600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.177566 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:22.309142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.309056 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9045c5cb-a562-4704-8bb9-5c63bf0c4585-kube-api-access-tmsw6\" (UniqueName: \"kubernetes.io/projected/9045c5cb-a562-4704-8bb9-5c63bf0c4585-kube-api-access-tmsw6\") pod \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\" (UID: \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\") " Nov 01 22:45:22.309316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.309292 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9045c5cb-a562-4704-8bb9-5c63bf0c4585-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045c5cb-a562-4704-8bb9-5c63bf0c4585-maas-api-service-ca\") pod \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\" (UID: \"9045c5cb-a562-4704-8bb9-5c63bf0c4585\") " Nov 01 22:45:22.310025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.309788 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9045c5cb-a562-4704-8bb9-5c63bf0c4585-maas-api-service-ca" pod "9045c5cb-a562-4704-8bb9-5c63bf0c4585" (UID: "9045c5cb-a562-4704-8bb9-5c63bf0c4585"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:22.310025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.309995 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9045c5cb-a562-4704-8bb9-5c63bf0c4585-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:22.311371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.311348 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9045c5cb-a562-4704-8bb9-5c63bf0c4585-kube-api-access-tmsw6" pod "9045c5cb-a562-4704-8bb9-5c63bf0c4585" (UID: "9045c5cb-a562-4704-8bb9-5c63bf0c4585"). InnerVolumeSpecName "kube-api-access-tmsw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:22.422843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.422801 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmsw6\" (UniqueName: \"kubernetes.io/projected/9045c5cb-a562-4704-8bb9-5c63bf0c4585-kube-api-access-tmsw6\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:22.644107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.644029 6287 generic.go:356] "Generic (PLEG): container finished" podID="afe868f0-c564-4a7b-a9d9-c9fbb83eacf3" containerID="2883be8cced56a999ccd07f760edfc8aacf84eeadd06007478b4384d4c6b0103" exitCode=0 Nov 01 22:45:22.644301 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.644143 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" event={"ID":"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3","Type":"ContainerDied","Data":"2883be8cced56a999ccd07f760edfc8aacf84eeadd06007478b4384d4c6b0103"} Nov 01 22:45:22.645781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.645758 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" Nov 01 22:45:22.645926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.645777 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh" event={"ID":"fdcadc29-3748-4c2b-80a2-60d946bca6e1","Type":"ContainerDied","Data":"e8a775e6d671ac16efbf3e87e9046212a8a43232ca2836a5ab33b2b6565fbfbc"} Nov 01 22:45:22.645926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.645834 6287 scope.go:122] "RemoveContainer" containerID="6e2d60c8384364ef63dc56f078803b33c0ad130467f147e4bdea4c0987934c90" Nov 01 22:45:22.651769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.651748 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" Nov 01 22:45:22.651891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.651789 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x" event={"ID":"9045c5cb-a562-4704-8bb9-5c63bf0c4585","Type":"ContainerDied","Data":"c2b7780b9cc969a10907a24584085446075331bb95ce1c2858600beabe7ad5e2"} Nov 01 22:45:22.665547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.665521 6287 scope.go:122] "RemoveContainer" containerID="22de36faaa5fef970efcae9bb1c8a6fc75f2e1d86bfc000e8e4f304d169a5b23" Nov 01 22:45:22.683353 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.683321 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh"] Nov 01 22:45:22.688054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.688029 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-ae0e3a-29892885-bp4rh"] Nov 01 22:45:22.710337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.710307 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x"] Nov 01 22:45:22.719193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:22.719157 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3d8490-29892885-hzd8x"] Nov 01 22:45:23.284288 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:23.284252 6287 patch_prober.go:29] interesting pod/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:45:23.284750 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:23.284316 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:23.475923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:23.475891 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" path="/var/lib/kubelet/pods/9045c5cb-a562-4704-8bb9-5c63bf0c4585/volumes" Nov 01 22:45:23.476400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:23.476384 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" path="/var/lib/kubelet/pods/fdcadc29-3748-4c2b-80a2-60d946bca6e1/volumes" Nov 01 22:45:23.893597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:23.893562 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:24.061526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.061490 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-kube-api-access-tbsq4\" (UniqueName: \"kubernetes.io/projected/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-kube-api-access-tbsq4\") pod \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " Nov 01 22:45:24.061682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.061589 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-xks-ca\") pod \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " Nov 01 22:45:24.061777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.061760 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-openshift-ca\") pod \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\" (UID: \"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3\") " Nov 01 22:45:24.062203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.062171 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-openshift-ca" pod "afe868f0-c564-4a7b-a9d9-c9fbb83eacf3" (UID: "afe868f0-c564-4a7b-a9d9-c9fbb83eacf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:24.062642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.062618 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.063845 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.063821 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-kube-api-access-tbsq4" pod "afe868f0-c564-4a7b-a9d9-c9fbb83eacf3" (UID: "afe868f0-c564-4a7b-a9d9-c9fbb83eacf3"). InnerVolumeSpecName "kube-api-access-tbsq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:24.064240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.064198 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-xks-ca" pod "afe868f0-c564-4a7b-a9d9-c9fbb83eacf3" (UID: "afe868f0-c564-4a7b-a9d9-c9fbb83eacf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:24.164427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.164396 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbsq4\" (UniqueName: \"kubernetes.io/projected/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-kube-api-access-tbsq4\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.164427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.164424 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.731997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.731965 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" Nov 01 22:45:24.732496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.732037 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl" event={"ID":"afe868f0-c564-4a7b-a9d9-c9fbb83eacf3","Type":"ContainerDied","Data":"41adcd7d00befa52e5d4432ceb4a38cf8169371a274b958c88d604254a12dedb"} Nov 01 22:45:24.732496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.732082 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41adcd7d00befa52e5d4432ceb4a38cf8169371a274b958c88d604254a12dedb" Nov 01 22:45:24.747435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.741726 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:45:24.751444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.750810 6287 generic.go:356] "Generic (PLEG): container finished" podID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerID="7ebada15c1c16daddb5b0258426b36981b407ff70243bb0a733048059580eb53" exitCode=0 Nov 01 22:45:24.751444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.750919 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" event={"ID":"1cf0787b-8f49-43c9-99dc-1805bec7e2ac","Type":"ContainerDied","Data":"7ebada15c1c16daddb5b0258426b36981b407ff70243bb0a733048059580eb53"} Nov 01 22:45:24.751444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.750975 6287 scope.go:122] "RemoveContainer" containerID="7ebada15c1c16daddb5b0258426b36981b407ff70243bb0a733048059580eb53" Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.808777 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-data\") pod \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.808863 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-envoy\") pod \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.808982 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-certs\") pod \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.809049 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-podinfo\") pod \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.809094 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-socket\") pod \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.809256 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istiod-ca-cert\") pod \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.809599 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-credential-socket\") pod \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.809670 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-kube-api-access-smt7l\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-kube-api-access-smt7l\") pod \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.809702 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-certs" pod "1cf0787b-8f49-43c9-99dc-1805bec7e2ac" (UID: "1cf0787b-8f49-43c9-99dc-1805bec7e2ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:24.809816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.809728 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-token\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-token\") pod \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\" (UID: \"1cf0787b-8f49-43c9-99dc-1805bec7e2ac\") " Nov 01 22:45:24.810636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.809999 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-data" pod "1cf0787b-8f49-43c9-99dc-1805bec7e2ac" (UID: "1cf0787b-8f49-43c9-99dc-1805bec7e2ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:24.812261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.812221 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.812261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.812252 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.814543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.814513 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istiod-ca-cert" pod "1cf0787b-8f49-43c9-99dc-1805bec7e2ac" (UID: "1cf0787b-8f49-43c9-99dc-1805bec7e2ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:24.816326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.816283 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-credential-socket" pod "1cf0787b-8f49-43c9-99dc-1805bec7e2ac" (UID: "1cf0787b-8f49-43c9-99dc-1805bec7e2ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:24.816675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.816644 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-socket" pod "1cf0787b-8f49-43c9-99dc-1805bec7e2ac" (UID: "1cf0787b-8f49-43c9-99dc-1805bec7e2ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:24.818448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.817665 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-podinfo" pod "1cf0787b-8f49-43c9-99dc-1805bec7e2ac" (UID: "1cf0787b-8f49-43c9-99dc-1805bec7e2ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:24.953904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.936300 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.953904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.936339 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.953904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.936363 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.953904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.936377 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.953904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.952667 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-token" pod "1cf0787b-8f49-43c9-99dc-1805bec7e2ac" (UID: "1cf0787b-8f49-43c9-99dc-1805bec7e2ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:24.972696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.972406 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-kube-api-access-smt7l" pod "1cf0787b-8f49-43c9-99dc-1805bec7e2ac" (UID: "1cf0787b-8f49-43c9-99dc-1805bec7e2ac"). InnerVolumeSpecName "kube-api-access-smt7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:24.979040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:24.978351 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-envoy" pod "1cf0787b-8f49-43c9-99dc-1805bec7e2ac" (UID: "1cf0787b-8f49-43c9-99dc-1805bec7e2ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:25.037680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.037648 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:25.037680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.037677 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:25.070342 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.037690 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smt7l\" (UniqueName: \"kubernetes.io/projected/1cf0787b-8f49-43c9-99dc-1805bec7e2ac-kube-api-access-smt7l\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:25.406967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.406271 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr"] Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407361 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407386 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbdba91-5443-4ec3-9d7e-efe7e397d20a" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407402 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407410 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407427 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407435 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407445 6287 cpu_manager.go:414] "containerMap: removing container" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407452 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407466 6287 cpu_manager.go:414] "containerMap: removing container" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407473 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407485 6287 cpu_manager.go:414] "containerMap: removing container" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407492 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407501 6287 cpu_manager.go:414] "containerMap: removing container" podUID="afe868f0-c564-4a7b-a9d9-c9fbb83eacf3" containerName="revoke-keys" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407508 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe868f0-c564-4a7b-a9d9-c9fbb83eacf3" containerName="revoke-keys" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407525 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407532 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407541 6287 cpu_manager.go:414] "containerMap: removing container" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407548 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407559 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407566 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407579 6287 cpu_manager.go:414] "containerMap: removing container" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407588 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407598 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407605 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407618 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407625 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407637 6287 cpu_manager.go:414] "containerMap: removing container" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407644 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407656 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407663 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407676 6287 cpu_manager.go:414] "containerMap: removing container" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407683 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407696 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407704 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407714 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407721 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407730 6287 cpu_manager.go:414] "containerMap: removing container" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407737 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407755 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407761 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407772 6287 cpu_manager.go:414] "containerMap: removing container" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407779 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407787 6287 cpu_manager.go:414] "containerMap: removing container" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407794 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407812 6287 cpu_manager.go:414] "containerMap: removing container" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407819 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407827 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407834 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407843 6287 cpu_manager.go:414] "containerMap: removing container" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407850 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407870 6287 cpu_manager.go:414] "containerMap: removing container" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407876 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407900 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407908 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407926 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407933 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407941 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407948 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407964 6287 cpu_manager.go:414] "containerMap: removing container" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.407970 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408164 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="452829c2-d248-41ef-a116-6810384b6951" containerName="main" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408183 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408190 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408200 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408207 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408217 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408226 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e2c051-67dd-41c7-8301-e5b5697c868e" containerName="main" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408234 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408243 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408251 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" containerName="istio-proxy" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408260 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408268 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408276 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408302 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408312 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408319 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408329 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408336 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408351 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408362 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408371 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408382 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fbbcb1-cd8b-497b-9588-e3015413a83e" containerName="main" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408396 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408411 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408421 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerName="cleanup" Nov 01 22:45:25.411666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.408430 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe868f0-c564-4a7b-a9d9-c9fbb83eacf3" containerName="revoke-keys" Nov 01 22:45:25.416436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.412729 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr"] Nov 01 22:45:25.416436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.412861 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.419795 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.419764 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:45:25.420258 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.420235 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:45:25.463538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.463506 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b72845ad-xp9hr\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.463768 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.463617 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqdx\" (UniqueName: \"kubernetes.io/projected/58e9298d-5d38-4620-815b-48910f1d13fd-kube-api-access-zkqdx\") pod \"maas-api-revoke-sub-b72845ad-xp9hr\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.463899 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.463769 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b72845ad-xp9hr\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.565035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.564995 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b72845ad-xp9hr\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.565269 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.565235 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b72845ad-xp9hr\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.565389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.565374 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqdx\" (UniqueName: \"kubernetes.io/projected/58e9298d-5d38-4620-815b-48910f1d13fd-kube-api-access-zkqdx\") pod \"maas-api-revoke-sub-b72845ad-xp9hr\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.566022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.565991 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b72845ad-xp9hr\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.567913 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.567895 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b72845ad-xp9hr\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.574846 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.574822 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqdx\" (UniqueName: \"kubernetes.io/projected/58e9298d-5d38-4620-815b-48910f1d13fd-kube-api-access-zkqdx\") pod \"maas-api-revoke-sub-b72845ad-xp9hr\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.737594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.737566 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:25.762526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.762499 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" Nov 01 22:45:25.762677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.762557 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h" event={"ID":"1cf0787b-8f49-43c9-99dc-1805bec7e2ac","Type":"ContainerDied","Data":"c2e5fc5ed4f2d56205eb5d600bece094502e083b6336ff38c1a84a777f4983b0"} Nov 01 22:45:25.765878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.765857 6287 generic.go:356] "Generic (PLEG): container finished" podID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerID="92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec" exitCode=0 Nov 01 22:45:25.765988 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.765913 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" event={"ID":"c18d254f-3b69-454e-a0b3-b8139db8b020","Type":"ContainerDied","Data":"92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec"} Nov 01 22:45:25.766382 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.766357 6287 scope.go:122] "RemoveContainer" containerID="92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec" Nov 01 22:45:25.768237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.768209 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:45:25.768323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.768276 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:45:25.768323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.768309 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:45:25.824264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.818541 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h"] Nov 01 22:45:25.824264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:25.821684 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-425870c1-openshift-default-65d9486fdf-dck6h"] Nov 01 22:45:26.056074 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:26.056042 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr"] Nov 01 22:45:26.056563 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:26.056525 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e9298d_5d38_4620_815b_48910f1d13fd.slice/crio-f0e505e803e77bab9cc60edc5d40846613accce6bde84d092c1b4fd59a3fb40a WatchSource:0}: Error finding container f0e505e803e77bab9cc60edc5d40846613accce6bde84d092c1b4fd59a3fb40a: Status 404 returned error can't find the container with id f0e505e803e77bab9cc60edc5d40846613accce6bde84d092c1b4fd59a3fb40a Nov 01 22:45:26.803650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:26.803612 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" event={"ID":"c18d254f-3b69-454e-a0b3-b8139db8b020","Type":"ContainerStarted","Data":"1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c"} Nov 01 22:45:26.806150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:26.806120 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" event={"ID":"58e9298d-5d38-4620-815b-48910f1d13fd","Type":"ContainerStarted","Data":"2fcb93bffadb7cac2617b5951c8f85635983d84ead9fac79692412e2d0ac514e"} Nov 01 22:45:26.806286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:26.806175 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" event={"ID":"58e9298d-5d38-4620-815b-48910f1d13fd","Type":"ContainerStarted","Data":"f0e505e803e77bab9cc60edc5d40846613accce6bde84d092c1b4fd59a3fb40a"} Nov 01 22:45:26.855716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:26.855663 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" podStartSLOduration=1.8556496359999999 podStartE2EDuration="1.855649636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:26.8533545 +0000 UTC m=+1339.501853734" watchObservedRunningTime="2026-11-01 22:45:26.855649636 +0000 UTC m=+1339.504149039" Nov 01 22:45:27.499505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.499473 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf0787b-8f49-43c9-99dc-1805bec7e2ac" path="/var/lib/kubelet/pods/1cf0787b-8f49-43c9-99dc-1805bec7e2ac/volumes" Nov 01 22:45:27.597526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.597486 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:45:27.597705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.597681 6287 patch_prober.go:29] interesting pod/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:45:27.597765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.597741 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:45:27.629103 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629072 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc"] Nov 01 22:45:27.629807 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629787 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerName="cleanup" Nov 01 22:45:27.629860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629810 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerName="cleanup" Nov 01 22:45:27.629860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629829 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerName="cleanup" Nov 01 22:45:27.629860 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629834 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerName="cleanup" Nov 01 22:45:27.629959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629862 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerName="cleanup" Nov 01 22:45:27.629959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629867 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerName="cleanup" Nov 01 22:45:27.629959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629879 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerName="cleanup" Nov 01 22:45:27.629959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629883 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerName="cleanup" Nov 01 22:45:27.629959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629898 6287 cpu_manager.go:414] "containerMap: removing container" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerName="cleanup" Nov 01 22:45:27.629959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629903 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8e2476-fb3a-496a-9f5a-ba753a927d97" containerName="cleanup" Nov 01 22:45:27.629959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629909 6287 cpu_manager.go:414] "containerMap: removing container" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerName="cleanup" Nov 01 22:45:27.629959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.629913 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerName="cleanup" Nov 01 22:45:27.630231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.630019 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6a9c68-a26b-4648-812b-1287c245cff0" containerName="cleanup" Nov 01 22:45:27.630231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.630028 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0db05bf-2b57-40d9-a059-5532c25fe5e3" containerName="cleanup" Nov 01 22:45:27.630231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.630037 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc7cc94-5af7-4618-8900-ce612c7a8ff8" containerName="cleanup" Nov 01 22:45:27.630231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.630045 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c8b3b3-4de9-430a-a5df-ad824446bead" containerName="cleanup" Nov 01 22:45:27.630231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.630053 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcadc29-3748-4c2b-80a2-60d946bca6e1" containerName="cleanup" Nov 01 22:45:27.630231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.630060 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77e5165-0098-48e8-8df6-ff4046a1d37b" containerName="cleanup" Nov 01 22:45:27.630231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.630066 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7ab1ac-2a1c-40cf-b39b-19981d199020" containerName="cleanup" Nov 01 22:45:27.630231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.630072 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="1427ec2e-5d9e-4f6a-b78f-a229cdd6bf93" containerName="cleanup" Nov 01 22:45:27.630231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.630078 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9045c5cb-a562-4704-8bb9-5c63bf0c4585" containerName="cleanup" Nov 01 22:45:27.634280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.634254 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6"] Nov 01 22:45:27.634456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.634370 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.639743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.639716 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.647687 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.647659 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc"] Nov 01 22:45:27.722733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.722693 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6"] Nov 01 22:45:27.731145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.729512 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tjl\" (UniqueName: \"kubernetes.io/projected/f3140a23-6ae1-4c8a-b7ee-512f12880378-kube-api-access-h9tjl\") pod \"maas-api-revoke-sub-2c6bb954-dh7c6\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.731145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.729733 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ec6a8a-72phc\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.731145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.729766 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ec6a8a-72phc\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.731145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.729795 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtdf\" (UniqueName: \"kubernetes.io/projected/5b726ae8-f84c-4208-b08c-cb8c34380870-kube-api-access-swtdf\") pod \"maas-api-revoke-sub-80ec6a8a-72phc\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.731145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.729864 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c6bb954-dh7c6\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.731145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.729998 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c6bb954-dh7c6\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.731145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.730185 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk"] Nov 01 22:45:27.731145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.730389 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" podUID="0f064454-385d-4336-abb8-288d642fe6fc" containerName="maas-api" containerID="cri-o://9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a" gracePeriod=30 Nov 01 22:45:27.820253 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.820222 6287 generic.go:356] "Generic (PLEG): container finished" podID="58e9298d-5d38-4620-815b-48910f1d13fd" containerID="2fcb93bffadb7cac2617b5951c8f85635983d84ead9fac79692412e2d0ac514e" exitCode=0 Nov 01 22:45:27.820639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.820307 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" event={"ID":"58e9298d-5d38-4620-815b-48910f1d13fd","Type":"ContainerDied","Data":"2fcb93bffadb7cac2617b5951c8f85635983d84ead9fac79692412e2d0ac514e"} Nov 01 22:45:27.831151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.831105 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tjl\" (UniqueName: \"kubernetes.io/projected/f3140a23-6ae1-4c8a-b7ee-512f12880378-kube-api-access-h9tjl\") pod \"maas-api-revoke-sub-2c6bb954-dh7c6\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.831282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.831250 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ec6a8a-72phc\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.831282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.831272 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ec6a8a-72phc\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.831396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.831297 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtdf\" (UniqueName: \"kubernetes.io/projected/5b726ae8-f84c-4208-b08c-cb8c34380870-kube-api-access-swtdf\") pod \"maas-api-revoke-sub-80ec6a8a-72phc\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.831396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.831334 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c6bb954-dh7c6\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.831489 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.831414 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c6bb954-dh7c6\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.832192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.832166 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c6bb954-dh7c6\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.832328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.832213 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ec6a8a-72phc\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.834028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.834007 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ec6a8a-72phc\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.834712 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.834691 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c6bb954-dh7c6\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.848642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.848608 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tjl\" (UniqueName: \"kubernetes.io/projected/f3140a23-6ae1-4c8a-b7ee-512f12880378-kube-api-access-h9tjl\") pod \"maas-api-revoke-sub-2c6bb954-dh7c6\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:27.849289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.849265 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtdf\" (UniqueName: \"kubernetes.io/projected/5b726ae8-f84c-4208-b08c-cb8c34380870-kube-api-access-swtdf\") pod \"maas-api-revoke-sub-80ec6a8a-72phc\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:27.965417 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:27.965379 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:28.003947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.003903 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:28.122252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.122211 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb"] Nov 01 22:45:28.168209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.168172 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b476e287-7l7sb"] Nov 01 22:45:28.318389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.318310 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt"] Nov 01 22:45:28.323165 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.322636 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3e228e-mjjvt"] Nov 01 22:45:28.531925 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.531887 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:45:28.613521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.613420 6287 patch_prober.go:29] interesting pod/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:45:28.613521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.613477 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:45:28.638907 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.638490 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc"] Nov 01 22:45:28.640677 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:28.640634 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b726ae8_f84c_4208_b08c_cb8c34380870.slice/crio-a27485b9db94ada35afdae6fd9c8aad3f032d45a79335c0be596fcc5b7f0a996 WatchSource:0}: Error finding container a27485b9db94ada35afdae6fd9c8aad3f032d45a79335c0be596fcc5b7f0a996: Status 404 returned error can't find the container with id a27485b9db94ada35afdae6fd9c8aad3f032d45a79335c0be596fcc5b7f0a996 Nov 01 22:45:28.715969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.715934 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f064454-385d-4336-abb8-288d642fe6fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f064454-385d-4336-abb8-288d642fe6fc-openshift-service-ca\") pod \"0f064454-385d-4336-abb8-288d642fe6fc\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " Nov 01 22:45:28.716227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.716074 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-maas-api-tls\") pod \"0f064454-385d-4336-abb8-288d642fe6fc\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " Nov 01 22:45:28.716319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.716256 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-metrics-certs\") pod \"0f064454-385d-4336-abb8-288d642fe6fc\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " Nov 01 22:45:28.716394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.716374 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f064454-385d-4336-abb8-288d642fe6fc-kube-api-access-r7mq5\" (UniqueName: \"kubernetes.io/projected/0f064454-385d-4336-abb8-288d642fe6fc-kube-api-access-r7mq5\") pod \"0f064454-385d-4336-abb8-288d642fe6fc\" (UID: \"0f064454-385d-4336-abb8-288d642fe6fc\") " Nov 01 22:45:28.717518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.716951 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f064454-385d-4336-abb8-288d642fe6fc-openshift-service-ca" pod "0f064454-385d-4336-abb8-288d642fe6fc" (UID: "0f064454-385d-4336-abb8-288d642fe6fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:28.719647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.719622 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-maas-api-tls" pod "0f064454-385d-4336-abb8-288d642fe6fc" (UID: "0f064454-385d-4336-abb8-288d642fe6fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:28.719800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.719780 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f064454-385d-4336-abb8-288d642fe6fc-kube-api-access-r7mq5" pod "0f064454-385d-4336-abb8-288d642fe6fc" (UID: "0f064454-385d-4336-abb8-288d642fe6fc"). InnerVolumeSpecName "kube-api-access-r7mq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:28.719970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.719954 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-metrics-certs" pod "0f064454-385d-4336-abb8-288d642fe6fc" (UID: "0f064454-385d-4336-abb8-288d642fe6fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:28.785145 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:28.781262 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3140a23_6ae1_4c8a_b7ee_512f12880378.slice/crio-543b57e4492a8a5f47a55f01605906f3a00680806381ed1fffbbfdad25ae10fa WatchSource:0}: Error finding container 543b57e4492a8a5f47a55f01605906f3a00680806381ed1fffbbfdad25ae10fa: Status 404 returned error can't find the container with id 543b57e4492a8a5f47a55f01605906f3a00680806381ed1fffbbfdad25ae10fa Nov 01 22:45:28.811182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.811151 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6"] Nov 01 22:45:28.821350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.819799 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-maas-api-tls\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:28.821350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.819827 6287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f064454-385d-4336-abb8-288d642fe6fc-metrics-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:28.821350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.819838 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7mq5\" (UniqueName: \"kubernetes.io/projected/0f064454-385d-4336-abb8-288d642fe6fc-kube-api-access-r7mq5\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:28.821350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.819847 6287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f064454-385d-4336-abb8-288d642fe6fc-openshift-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:28.834718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.834686 6287 generic.go:356] "Generic (PLEG): container finished" podID="0f064454-385d-4336-abb8-288d642fe6fc" containerID="9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a" exitCode=0 Nov 01 22:45:28.834867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.834743 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" Nov 01 22:45:28.834867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.834762 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" event={"ID":"0f064454-385d-4336-abb8-288d642fe6fc","Type":"ContainerDied","Data":"9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a"} Nov 01 22:45:28.834867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.834816 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk" event={"ID":"0f064454-385d-4336-abb8-288d642fe6fc","Type":"ContainerDied","Data":"34376cff6258d858c67bbc9a64cce952b01c5965fe13ef12898c623fd11b0532"} Nov 01 22:45:28.834867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.834861 6287 scope.go:122] "RemoveContainer" containerID="9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a" Nov 01 22:45:28.837196 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.837035 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" event={"ID":"f3140a23-6ae1-4c8a-b7ee-512f12880378","Type":"ContainerStarted","Data":"543b57e4492a8a5f47a55f01605906f3a00680806381ed1fffbbfdad25ae10fa"} Nov 01 22:45:28.844257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.844210 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" event={"ID":"5b726ae8-f84c-4208-b08c-cb8c34380870","Type":"ContainerStarted","Data":"a27485b9db94ada35afdae6fd9c8aad3f032d45a79335c0be596fcc5b7f0a996"} Nov 01 22:45:28.850371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.850345 6287 scope.go:122] "RemoveContainer" containerID="9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a" Nov 01 22:45:28.850801 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:28.850766 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a\": container with ID starting with 9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a not found: ID does not exist" containerID="9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a" Nov 01 22:45:28.851006 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.850813 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a"} err="failed to get container status \"9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a\": rpc error: code = NotFound desc = could not find container \"9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a\": container with ID starting with 9b557f51bc93da2c3a279875b51f267eabd82917da8bec177f6d6942f3787b5a not found: ID does not exist" Nov 01 22:45:28.866521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.866455 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk"] Nov 01 22:45:28.871769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:28.871733 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e228e-655579484b-ftswk"] Nov 01 22:45:29.077025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.076641 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:29.123763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.123678 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e9298d-5d38-4620-815b-48910f1d13fd-kube-api-access-zkqdx\" (UniqueName: \"kubernetes.io/projected/58e9298d-5d38-4620-815b-48910f1d13fd-kube-api-access-zkqdx\") pod \"58e9298d-5d38-4620-815b-48910f1d13fd\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " Nov 01 22:45:29.123763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.123723 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-xks-ca\") pod \"58e9298d-5d38-4620-815b-48910f1d13fd\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " Nov 01 22:45:29.123763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.123745 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-openshift-ca\") pod \"58e9298d-5d38-4620-815b-48910f1d13fd\" (UID: \"58e9298d-5d38-4620-815b-48910f1d13fd\") " Nov 01 22:45:29.124799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.124769 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-openshift-ca" pod "58e9298d-5d38-4620-815b-48910f1d13fd" (UID: "58e9298d-5d38-4620-815b-48910f1d13fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:29.127914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.127870 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-xks-ca" pod "58e9298d-5d38-4620-815b-48910f1d13fd" (UID: "58e9298d-5d38-4620-815b-48910f1d13fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:29.128052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.128030 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e9298d-5d38-4620-815b-48910f1d13fd-kube-api-access-zkqdx" pod "58e9298d-5d38-4620-815b-48910f1d13fd" (UID: "58e9298d-5d38-4620-815b-48910f1d13fd"). InnerVolumeSpecName "kube-api-access-zkqdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:29.226448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.226412 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqdx\" (UniqueName: \"kubernetes.io/projected/58e9298d-5d38-4620-815b-48910f1d13fd-kube-api-access-zkqdx\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:29.226607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.226449 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:29.226607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.226467 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e9298d-5d38-4620-815b-48910f1d13fd-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:29.598548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.598102 6287 patch_prober.go:29] interesting pod/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:45:29.598548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.598173 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:45:29.636076 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.636037 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f064454-385d-4336-abb8-288d642fe6fc" path="/var/lib/kubelet/pods/0f064454-385d-4336-abb8-288d642fe6fc/volumes" Nov 01 22:45:29.637834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.637795 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa" path="/var/lib/kubelet/pods/4db8a3e5-0cb5-4beb-bc05-a2e6f5e885aa/volumes" Nov 01 22:45:29.640830 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.640792 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e98c1d-7b61-49ad-9836-542804accd90" path="/var/lib/kubelet/pods/f6e98c1d-7b61-49ad-9836-542804accd90/volumes" Nov 01 22:45:29.872374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.869762 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" event={"ID":"58e9298d-5d38-4620-815b-48910f1d13fd","Type":"ContainerDied","Data":"f0e505e803e77bab9cc60edc5d40846613accce6bde84d092c1b4fd59a3fb40a"} Nov 01 22:45:29.872374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.869825 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e505e803e77bab9cc60edc5d40846613accce6bde84d092c1b4fd59a3fb40a" Nov 01 22:45:29.872374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.869905 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr" Nov 01 22:45:29.876956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.875785 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" event={"ID":"5b726ae8-f84c-4208-b08c-cb8c34380870","Type":"ContainerStarted","Data":"fca402f8081c9033150942ec78e8d6633d233b4596aa8bd72416c5f660b8dcb9"} Nov 01 22:45:29.888632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.888600 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" event={"ID":"f3140a23-6ae1-4c8a-b7ee-512f12880378","Type":"ContainerStarted","Data":"d618757e2bb869b479653daeee40c2cb707efde127d66883f0fa462983888f6e"} Nov 01 22:45:29.914277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.914220 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" podStartSLOduration=2.914200835 podStartE2EDuration="2.914200835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:29.905676641 +0000 UTC m=+1342.554175882" watchObservedRunningTime="2026-11-01 22:45:29.914200835 +0000 UTC m=+1342.562700071" Nov 01 22:45:29.939607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.939556 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" podStartSLOduration=2.939542149 podStartE2EDuration="2.939542149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:29.938366578 +0000 UTC m=+1342.586865813" watchObservedRunningTime="2026-11-01 22:45:29.939542149 +0000 UTC m=+1342.588041383" Nov 01 22:45:29.964414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.964360 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl"] Nov 01 22:45:29.968873 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:29.968844 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-10a1d9-cd2wl"] Nov 01 22:45:30.006706 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.006668 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" podUID="8ac2dd5a-11ff-408d-96f0-dac764508392" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:45:30.160651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.160563 6287 scope.go:122] "RemoveContainer" containerID="c9a0258f97eb41acf4a79679a30b6af4519ae5730444915e57529d437707ffde" Nov 01 22:45:30.203381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.203339 6287 scope.go:122] "RemoveContainer" containerID="db7bd1adbc057b82eef9686bb7f57e8116c0480d56a590c8e1a42c67988cd5ac" Nov 01 22:45:30.215627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.215601 6287 scope.go:122] "RemoveContainer" containerID="67a4fd95e412ba893f0842d8cbb8493c6473416ed59aaba7d857b46bf3fdd143" Nov 01 22:45:30.597803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.597768 6287 patch_prober.go:29] interesting pod/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:45:30.598022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.597840 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:45:30.906006 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.905972 6287 generic.go:356] "Generic (PLEG): container finished" podID="5b726ae8-f84c-4208-b08c-cb8c34380870" containerID="fca402f8081c9033150942ec78e8d6633d233b4596aa8bd72416c5f660b8dcb9" exitCode=0 Nov 01 22:45:30.906480 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.906052 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" event={"ID":"5b726ae8-f84c-4208-b08c-cb8c34380870","Type":"ContainerDied","Data":"fca402f8081c9033150942ec78e8d6633d233b4596aa8bd72416c5f660b8dcb9"} Nov 01 22:45:30.907757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.907734 6287 generic.go:356] "Generic (PLEG): container finished" podID="f3140a23-6ae1-4c8a-b7ee-512f12880378" containerID="d618757e2bb869b479653daeee40c2cb707efde127d66883f0fa462983888f6e" exitCode=0 Nov 01 22:45:30.907870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:30.907802 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" event={"ID":"f3140a23-6ae1-4c8a-b7ee-512f12880378","Type":"ContainerDied","Data":"d618757e2bb869b479653daeee40c2cb707efde127d66883f0fa462983888f6e"} Nov 01 22:45:31.041785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:31.041749 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" podUID="9644b916-6603-48dc-b6e8-993bc7cec7f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:45:31.500085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:31.500048 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe868f0-c564-4a7b-a9d9-c9fbb83eacf3" path="/var/lib/kubelet/pods/afe868f0-c564-4a7b-a9d9-c9fbb83eacf3/volumes" Nov 01 22:45:31.600629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:31.600589 6287 patch_prober.go:29] interesting pod/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 22:45:31.600797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:31.600666 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 22:45:32.226686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.226654 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:32.314616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.314580 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:32.337508 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.337475 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b726ae8-f84c-4208-b08c-cb8c34380870-kube-api-access-swtdf\" (UniqueName: \"kubernetes.io/projected/5b726ae8-f84c-4208-b08c-cb8c34380870-kube-api-access-swtdf\") pod \"5b726ae8-f84c-4208-b08c-cb8c34380870\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " Nov 01 22:45:32.337677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.337643 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3140a23-6ae1-4c8a-b7ee-512f12880378-kube-api-access-h9tjl\" (UniqueName: \"kubernetes.io/projected/f3140a23-6ae1-4c8a-b7ee-512f12880378-kube-api-access-h9tjl\") pod \"f3140a23-6ae1-4c8a-b7ee-512f12880378\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " Nov 01 22:45:32.337754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.337673 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-xks-ca\") pod \"5b726ae8-f84c-4208-b08c-cb8c34380870\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " Nov 01 22:45:32.337754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.337733 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-xks-ca\") pod \"f3140a23-6ae1-4c8a-b7ee-512f12880378\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " Nov 01 22:45:32.337897 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.337839 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-openshift-ca\") pod \"f3140a23-6ae1-4c8a-b7ee-512f12880378\" (UID: \"f3140a23-6ae1-4c8a-b7ee-512f12880378\") " Nov 01 22:45:32.338050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.337898 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-openshift-ca\") pod \"5b726ae8-f84c-4208-b08c-cb8c34380870\" (UID: \"5b726ae8-f84c-4208-b08c-cb8c34380870\") " Nov 01 22:45:32.339069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.339041 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-openshift-ca" pod "5b726ae8-f84c-4208-b08c-cb8c34380870" (UID: "5b726ae8-f84c-4208-b08c-cb8c34380870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:32.339176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.339088 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-openshift-ca" pod "f3140a23-6ae1-4c8a-b7ee-512f12880378" (UID: "f3140a23-6ae1-4c8a-b7ee-512f12880378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:32.341236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.341208 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-xks-ca" pod "5b726ae8-f84c-4208-b08c-cb8c34380870" (UID: "5b726ae8-f84c-4208-b08c-cb8c34380870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:32.341360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.341334 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b726ae8-f84c-4208-b08c-cb8c34380870-kube-api-access-swtdf" pod "5b726ae8-f84c-4208-b08c-cb8c34380870" (UID: "5b726ae8-f84c-4208-b08c-cb8c34380870"). InnerVolumeSpecName "kube-api-access-swtdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:32.341706 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.341672 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-xks-ca" pod "f3140a23-6ae1-4c8a-b7ee-512f12880378" (UID: "f3140a23-6ae1-4c8a-b7ee-512f12880378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:32.342009 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.341978 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3140a23-6ae1-4c8a-b7ee-512f12880378-kube-api-access-h9tjl" pod "f3140a23-6ae1-4c8a-b7ee-512f12880378" (UID: "f3140a23-6ae1-4c8a-b7ee-512f12880378"). InnerVolumeSpecName "kube-api-access-h9tjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:32.440347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.440255 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9tjl\" (UniqueName: \"kubernetes.io/projected/f3140a23-6ae1-4c8a-b7ee-512f12880378-kube-api-access-h9tjl\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:32.440347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.440296 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:32.440347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.440312 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:32.440347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.440326 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3140a23-6ae1-4c8a-b7ee-512f12880378-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:32.440347 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.440340 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b726ae8-f84c-4208-b08c-cb8c34380870-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:32.441286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.440354 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swtdf\" (UniqueName: \"kubernetes.io/projected/5b726ae8-f84c-4208-b08c-cb8c34380870-kube-api-access-swtdf\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:32.601820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.601791 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:45:32.602317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.602290 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:45:32.602853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.602834 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:45:32.978308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.978275 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" Nov 01 22:45:32.978529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.978305 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6" event={"ID":"f3140a23-6ae1-4c8a-b7ee-512f12880378","Type":"ContainerDied","Data":"543b57e4492a8a5f47a55f01605906f3a00680806381ed1fffbbfdad25ae10fa"} Nov 01 22:45:32.978529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.978364 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="543b57e4492a8a5f47a55f01605906f3a00680806381ed1fffbbfdad25ae10fa" Nov 01 22:45:32.980184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.980154 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" event={"ID":"5b726ae8-f84c-4208-b08c-cb8c34380870","Type":"ContainerDied","Data":"a27485b9db94ada35afdae6fd9c8aad3f032d45a79335c0be596fcc5b7f0a996"} Nov 01 22:45:32.980322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.980206 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a27485b9db94ada35afdae6fd9c8aad3f032d45a79335c0be596fcc5b7f0a996" Nov 01 22:45:32.980423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:32.980354 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc" Nov 01 22:45:35.268847 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.268818 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:45:35.298834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.298807 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:45:35.420008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.419975 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9644b916-6603-48dc-b6e8-993bc7cec7f8-kube-api-access-7rtt7\" (UniqueName: \"kubernetes.io/projected/9644b916-6603-48dc-b6e8-993bc7cec7f8-kube-api-access-7rtt7\") pod \"9644b916-6603-48dc-b6e8-993bc7cec7f8\" (UID: \"9644b916-6603-48dc-b6e8-993bc7cec7f8\") " Nov 01 22:45:35.420254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.420199 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9644b916-6603-48dc-b6e8-993bc7cec7f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9644b916-6603-48dc-b6e8-993bc7cec7f8-plugins-config-volume\") pod \"9644b916-6603-48dc-b6e8-993bc7cec7f8\" (UID: \"9644b916-6603-48dc-b6e8-993bc7cec7f8\") " Nov 01 22:45:35.420341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.420281 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac2dd5a-11ff-408d-96f0-dac764508392-kube-api-access-zjxlf\" (UniqueName: \"kubernetes.io/projected/8ac2dd5a-11ff-408d-96f0-dac764508392-kube-api-access-zjxlf\") pod \"8ac2dd5a-11ff-408d-96f0-dac764508392\" (UID: \"8ac2dd5a-11ff-408d-96f0-dac764508392\") " Nov 01 22:45:35.420341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.420315 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac2dd5a-11ff-408d-96f0-dac764508392-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2dd5a-11ff-408d-96f0-dac764508392-plugins-config-volume\") pod \"8ac2dd5a-11ff-408d-96f0-dac764508392\" (UID: \"8ac2dd5a-11ff-408d-96f0-dac764508392\") " Nov 01 22:45:35.420664 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.420637 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9644b916-6603-48dc-b6e8-993bc7cec7f8-plugins-config-volume" pod "9644b916-6603-48dc-b6e8-993bc7cec7f8" (UID: "9644b916-6603-48dc-b6e8-993bc7cec7f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:35.420948 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.420917 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac2dd5a-11ff-408d-96f0-dac764508392-plugins-config-volume" pod "8ac2dd5a-11ff-408d-96f0-dac764508392" (UID: "8ac2dd5a-11ff-408d-96f0-dac764508392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:35.421091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.421071 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9644b916-6603-48dc-b6e8-993bc7cec7f8-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:35.421182 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.421099 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2dd5a-11ff-408d-96f0-dac764508392-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:35.422349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.422331 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9644b916-6603-48dc-b6e8-993bc7cec7f8-kube-api-access-7rtt7" pod "9644b916-6603-48dc-b6e8-993bc7cec7f8" (UID: "9644b916-6603-48dc-b6e8-993bc7cec7f8"). InnerVolumeSpecName "kube-api-access-7rtt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:35.422408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.422360 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac2dd5a-11ff-408d-96f0-dac764508392-kube-api-access-zjxlf" pod "8ac2dd5a-11ff-408d-96f0-dac764508392" (UID: "8ac2dd5a-11ff-408d-96f0-dac764508392"). InnerVolumeSpecName "kube-api-access-zjxlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:35.522742 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.522669 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjxlf\" (UniqueName: \"kubernetes.io/projected/8ac2dd5a-11ff-408d-96f0-dac764508392-kube-api-access-zjxlf\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:35.522742 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.522700 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtt7\" (UniqueName: \"kubernetes.io/projected/9644b916-6603-48dc-b6e8-993bc7cec7f8-kube-api-access-7rtt7\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:35.575878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.575847 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6"] Nov 01 22:45:35.576531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576514 6287 cpu_manager.go:414] "containerMap: removing container" podUID="9644b916-6603-48dc-b6e8-993bc7cec7f8" containerName="payload-pre-processing" Nov 01 22:45:35.576531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576532 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9644b916-6603-48dc-b6e8-993bc7cec7f8" containerName="payload-pre-processing" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576550 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5b726ae8-f84c-4208-b08c-cb8c34380870" containerName="revoke-keys" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576556 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b726ae8-f84c-4208-b08c-cb8c34380870" containerName="revoke-keys" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576574 6287 cpu_manager.go:414] "containerMap: removing container" podUID="58e9298d-5d38-4620-815b-48910f1d13fd" containerName="revoke-keys" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576580 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e9298d-5d38-4620-815b-48910f1d13fd" containerName="revoke-keys" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576589 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0f064454-385d-4336-abb8-288d642fe6fc" containerName="maas-api" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576594 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f064454-385d-4336-abb8-288d642fe6fc" containerName="maas-api" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576625 6287 cpu_manager.go:414] "containerMap: removing container" podUID="8ac2dd5a-11ff-408d-96f0-dac764508392" containerName="payload-processing" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576630 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac2dd5a-11ff-408d-96f0-dac764508392" containerName="payload-processing" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576637 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f3140a23-6ae1-4c8a-b7ee-512f12880378" containerName="revoke-keys" Nov 01 22:45:35.576658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576642 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3140a23-6ae1-4c8a-b7ee-512f12880378" containerName="revoke-keys" Nov 01 22:45:35.576973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576734 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f064454-385d-4336-abb8-288d642fe6fc" containerName="maas-api" Nov 01 22:45:35.576973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576746 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac2dd5a-11ff-408d-96f0-dac764508392" containerName="payload-processing" Nov 01 22:45:35.576973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576751 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3140a23-6ae1-4c8a-b7ee-512f12880378" containerName="revoke-keys" Nov 01 22:45:35.576973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576761 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9644b916-6603-48dc-b6e8-993bc7cec7f8" containerName="payload-pre-processing" Nov 01 22:45:35.576973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576768 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b726ae8-f84c-4208-b08c-cb8c34380870" containerName="revoke-keys" Nov 01 22:45:35.576973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.576777 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e9298d-5d38-4620-815b-48910f1d13fd" containerName="revoke-keys" Nov 01 22:45:35.580590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.580573 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.589204 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.589178 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6"] Nov 01 22:45:35.589526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.589502 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:45:35.590928 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.590749 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:45:35.590928 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.590783 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:45:35.590928 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.590808 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:45:35.725753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.725717 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skn2c\" (UniqueName: \"kubernetes.io/projected/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-kube-api-access-skn2c\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.725943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.725919 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.726018 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.725978 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.827185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.827082 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skn2c\" (UniqueName: \"kubernetes.io/projected/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-kube-api-access-skn2c\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.827323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.827269 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.827323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.827305 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.827886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.827865 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.829785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.829765 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.843044 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.843023 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skn2c\" (UniqueName: \"kubernetes.io/projected/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-kube-api-access-skn2c\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:35.901887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:35.901865 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:36.008264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.008232 6287 generic.go:356] "Generic (PLEG): container finished" podID="8ac2dd5a-11ff-408d-96f0-dac764508392" containerID="6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785" exitCode=0 Nov 01 22:45:36.008405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.008282 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" Nov 01 22:45:36.008700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.008484 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" event={"ID":"8ac2dd5a-11ff-408d-96f0-dac764508392","Type":"ContainerDied","Data":"6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785"} Nov 01 22:45:36.008700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.008545 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns" event={"ID":"8ac2dd5a-11ff-408d-96f0-dac764508392","Type":"ContainerDied","Data":"387a813ee4fb626b6433edb8a2d1eb2ca50bc3edd69c3030c27d8b28baffe6fd"} Nov 01 22:45:36.008700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.008576 6287 scope.go:122] "RemoveContainer" containerID="6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785" Nov 01 22:45:36.010829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.010805 6287 generic.go:356] "Generic (PLEG): container finished" podID="9644b916-6603-48dc-b6e8-993bc7cec7f8" containerID="3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5" exitCode=0 Nov 01 22:45:36.010928 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.010912 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" event={"ID":"9644b916-6603-48dc-b6e8-993bc7cec7f8","Type":"ContainerDied","Data":"3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5"} Nov 01 22:45:36.010973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.010957 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" event={"ID":"9644b916-6603-48dc-b6e8-993bc7cec7f8","Type":"ContainerDied","Data":"86a4d20f4f7ddc092607f8e3392353302700af067244a51a968c1248e5c93c66"} Nov 01 22:45:36.011040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.011026 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw" Nov 01 22:45:36.024250 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.024162 6287 scope.go:122] "RemoveContainer" containerID="6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785" Nov 01 22:45:36.024615 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:36.024472 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785\": container with ID starting with 6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785 not found: ID does not exist" containerID="6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785" Nov 01 22:45:36.024615 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.024508 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785"} err="failed to get container status \"6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785\": rpc error: code = NotFound desc = could not find container \"6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785\": container with ID starting with 6ffda6469117c79818b6b111659d6e3abc100cb86eff100348e437b4daf15785 not found: ID does not exist" Nov 01 22:45:36.024615 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.024530 6287 scope.go:122] "RemoveContainer" containerID="3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5" Nov 01 22:45:36.036425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.036402 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns"] Nov 01 22:45:36.041950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.041922 6287 scope.go:122] "RemoveContainer" containerID="3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5" Nov 01 22:45:36.042458 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:36.042396 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5\": container with ID starting with 3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5 not found: ID does not exist" containerID="3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5" Nov 01 22:45:36.042458 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.042427 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5"} err="failed to get container status \"3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5\": rpc error: code = NotFound desc = could not find container \"3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5\": container with ID starting with 3d03a71ddc00a7062a041658b40fadb1918c32f379cc18a38380c266ba51f1b5 not found: ID does not exist" Nov 01 22:45:36.046800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.046776 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b42cc4-59d647b6d7-8kvns"] Nov 01 22:45:36.070154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.070125 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw"] Nov 01 22:45:36.074323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.074300 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b42cc4-6c59c87fcf-pfkgw"] Nov 01 22:45:36.189184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:36.189150 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6"] Nov 01 22:45:36.189633 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:36.189605 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c9b23d_a634_4275_9bbc_eebe57d2fbd7.slice/crio-d2d79708cf4d4cd869eef7f2f39bac9052c1e01ed761d772698aa8f5ec60aea5 WatchSource:0}: Error finding container d2d79708cf4d4cd869eef7f2f39bac9052c1e01ed761d772698aa8f5ec60aea5: Status 404 returned error can't find the container with id d2d79708cf4d4cd869eef7f2f39bac9052c1e01ed761d772698aa8f5ec60aea5 Nov 01 22:45:37.030026 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.029985 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" event={"ID":"11c9b23d-a634-4275-9bbc-eebe57d2fbd7","Type":"ContainerStarted","Data":"7997d1aaabfd6c03603a56192afc17728c2a3c4860c13ca7ee386d8c64a7fa71"} Nov 01 22:45:37.030616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.030050 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" event={"ID":"11c9b23d-a634-4275-9bbc-eebe57d2fbd7","Type":"ContainerStarted","Data":"d2d79708cf4d4cd869eef7f2f39bac9052c1e01ed761d772698aa8f5ec60aea5"} Nov 01 22:45:37.066415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.066351 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" podStartSLOduration=2.066331219 podStartE2EDuration="2.066331219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:37.062499288 +0000 UTC m=+1349.710998523" watchObservedRunningTime="2026-11-01 22:45:37.066331219 +0000 UTC m=+1349.714830455" Nov 01 22:45:37.234050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.234019 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj"] Nov 01 22:45:37.252094 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.252061 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.256037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.256006 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj"] Nov 01 22:45:37.357545 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.357469 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2555-q74fj\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.357687 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.357613 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27j2d\" (UniqueName: \"kubernetes.io/projected/5dde28fb-3925-4b15-9bb9-76c654c5b88c-kube-api-access-27j2d\") pod \"maas-api-revoke-keys-e2e-api-b-8f2555-q74fj\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.357752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.357736 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2555-q74fj\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.459374 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.459345 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27j2d\" (UniqueName: \"kubernetes.io/projected/5dde28fb-3925-4b15-9bb9-76c654c5b88c-kube-api-access-27j2d\") pod \"maas-api-revoke-keys-e2e-api-b-8f2555-q74fj\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.459517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.459445 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2555-q74fj\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.459564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.459530 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2555-q74fj\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.460198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.460164 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2555-q74fj\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.461910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.461876 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f2555-q74fj\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.468261 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.468235 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27j2d\" (UniqueName: \"kubernetes.io/projected/5dde28fb-3925-4b15-9bb9-76c654c5b88c-kube-api-access-27j2d\") pod \"maas-api-revoke-keys-e2e-api-b-8f2555-q74fj\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.473594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.473570 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac2dd5a-11ff-408d-96f0-dac764508392" path="/var/lib/kubelet/pods/8ac2dd5a-11ff-408d-96f0-dac764508392/volumes" Nov 01 22:45:37.473972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.473958 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9644b916-6603-48dc-b6e8-993bc7cec7f8" path="/var/lib/kubelet/pods/9644b916-6603-48dc-b6e8-993bc7cec7f8/volumes" Nov 01 22:45:37.578512 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.578481 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:37.872445 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:37.872415 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dde28fb_3925_4b15_9bb9_76c654c5b88c.slice/crio-2fbdbff245eca25c9e175c5fc8144f7eb9534a480c21c31278751d5eb6aa5ffd WatchSource:0}: Error finding container 2fbdbff245eca25c9e175c5fc8144f7eb9534a480c21c31278751d5eb6aa5ffd: Status 404 returned error can't find the container with id 2fbdbff245eca25c9e175c5fc8144f7eb9534a480c21c31278751d5eb6aa5ffd Nov 01 22:45:37.873781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:37.873760 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj"] Nov 01 22:45:38.040569 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:38.040538 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" event={"ID":"5dde28fb-3925-4b15-9bb9-76c654c5b88c","Type":"ContainerStarted","Data":"2fbdbff245eca25c9e175c5fc8144f7eb9534a480c21c31278751d5eb6aa5ffd"} Nov 01 22:45:38.042562 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:38.042514 6287 generic.go:356] "Generic (PLEG): container finished" podID="11c9b23d-a634-4275-9bbc-eebe57d2fbd7" containerID="7997d1aaabfd6c03603a56192afc17728c2a3c4860c13ca7ee386d8c64a7fa71" exitCode=0 Nov 01 22:45:38.042688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:38.042627 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" event={"ID":"11c9b23d-a634-4275-9bbc-eebe57d2fbd7","Type":"ContainerDied","Data":"7997d1aaabfd6c03603a56192afc17728c2a3c4860c13ca7ee386d8c64a7fa71"} Nov 01 22:45:39.057848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.057756 6287 generic.go:356] "Generic (PLEG): container finished" podID="5dde28fb-3925-4b15-9bb9-76c654c5b88c" containerID="2a961ae10f0630ffde89a5adcbb9d5582bc18929d650784b54b09968f5396aff" exitCode=0 Nov 01 22:45:39.057848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.057829 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" event={"ID":"5dde28fb-3925-4b15-9bb9-76c654c5b88c","Type":"ContainerDied","Data":"2a961ae10f0630ffde89a5adcbb9d5582bc18929d650784b54b09968f5396aff"} Nov 01 22:45:39.221225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.221197 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:39.302164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.285529 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-kube-api-access-skn2c\" (UniqueName: \"kubernetes.io/projected/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-kube-api-access-skn2c\") pod \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " Nov 01 22:45:39.302164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.285644 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-xks-ca\") pod \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " Nov 01 22:45:39.302164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.285683 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-openshift-ca\") pod \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\" (UID: \"11c9b23d-a634-4275-9bbc-eebe57d2fbd7\") " Nov 01 22:45:39.302164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.286363 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-openshift-ca" pod "11c9b23d-a634-4275-9bbc-eebe57d2fbd7" (UID: "11c9b23d-a634-4275-9bbc-eebe57d2fbd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:39.302164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.286507 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:39.302164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.288808 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-kube-api-access-skn2c" pod "11c9b23d-a634-4275-9bbc-eebe57d2fbd7" (UID: "11c9b23d-a634-4275-9bbc-eebe57d2fbd7"). InnerVolumeSpecName "kube-api-access-skn2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:39.302164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.289665 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-xks-ca" pod "11c9b23d-a634-4275-9bbc-eebe57d2fbd7" (UID: "11c9b23d-a634-4275-9bbc-eebe57d2fbd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:39.388236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.388166 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skn2c\" (UniqueName: \"kubernetes.io/projected/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-kube-api-access-skn2c\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:39.388236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.388190 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c9b23d-a634-4275-9bbc-eebe57d2fbd7-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:39.787041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.784309 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8"] Nov 01 22:45:39.787041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.785954 6287 cpu_manager.go:414] "containerMap: removing container" podUID="11c9b23d-a634-4275-9bbc-eebe57d2fbd7" containerName="revoke-keys" Nov 01 22:45:39.787041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.785977 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c9b23d-a634-4275-9bbc-eebe57d2fbd7" containerName="revoke-keys" Nov 01 22:45:39.787041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.786338 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c9b23d-a634-4275-9bbc-eebe57d2fbd7" containerName="revoke-keys" Nov 01 22:45:39.797140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.797091 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:39.807019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.805879 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8"] Nov 01 22:45:39.908183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.908146 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:39.908366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.908343 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmkk\" (UniqueName: \"kubernetes.io/projected/2a4657fe-6a6b-4478-bcee-27b745b82cf1-kube-api-access-szmkk\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:39.908877 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:39.908397 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:40.009708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.009684 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:40.009861 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.009769 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmkk\" (UniqueName: \"kubernetes.io/projected/2a4657fe-6a6b-4478-bcee-27b745b82cf1-kube-api-access-szmkk\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:40.009861 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.009804 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:40.010375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.010353 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:40.012089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.012069 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:40.017818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.017799 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmkk\" (UniqueName: \"kubernetes.io/projected/2a4657fe-6a6b-4478-bcee-27b745b82cf1-kube-api-access-szmkk\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:40.067572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.067546 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" Nov 01 22:45:40.067972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.067567 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6" event={"ID":"11c9b23d-a634-4275-9bbc-eebe57d2fbd7","Type":"ContainerDied","Data":"d2d79708cf4d4cd869eef7f2f39bac9052c1e01ed761d772698aa8f5ec60aea5"} Nov 01 22:45:40.067972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.067613 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d79708cf4d4cd869eef7f2f39bac9052c1e01ed761d772698aa8f5ec60aea5" Nov 01 22:45:40.121008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.120970 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:40.329357 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.329326 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:40.418488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.418328 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-xks-ca\") pod \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " Nov 01 22:45:40.418488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.418460 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-openshift-ca\") pod \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " Nov 01 22:45:40.418903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.418850 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-openshift-ca" pod "5dde28fb-3925-4b15-9bb9-76c654c5b88c" (UID: "5dde28fb-3925-4b15-9bb9-76c654c5b88c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:40.419057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.418915 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dde28fb-3925-4b15-9bb9-76c654c5b88c-kube-api-access-27j2d\" (UniqueName: \"kubernetes.io/projected/5dde28fb-3925-4b15-9bb9-76c654c5b88c-kube-api-access-27j2d\") pod \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\" (UID: \"5dde28fb-3925-4b15-9bb9-76c654c5b88c\") " Nov 01 22:45:40.419887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.419660 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:40.424561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.424533 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dde28fb-3925-4b15-9bb9-76c654c5b88c-kube-api-access-27j2d" pod "5dde28fb-3925-4b15-9bb9-76c654c5b88c" (UID: "5dde28fb-3925-4b15-9bb9-76c654c5b88c"). InnerVolumeSpecName "kube-api-access-27j2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:40.425203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.425182 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-xks-ca" pod "5dde28fb-3925-4b15-9bb9-76c654c5b88c" (UID: "5dde28fb-3925-4b15-9bb9-76c654c5b88c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:40.460858 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:45:40.460766 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4657fe_6a6b_4478_bcee_27b745b82cf1.slice/crio-45570529d5dfeda166e1135874d0b07e01e8c78f2f9f64f4d8798da813eaaf4a WatchSource:0}: Error finding container 45570529d5dfeda166e1135874d0b07e01e8c78f2f9f64f4d8798da813eaaf4a: Status 404 returned error can't find the container with id 45570529d5dfeda166e1135874d0b07e01e8c78f2f9f64f4d8798da813eaaf4a Nov 01 22:45:40.462516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.462410 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8"] Nov 01 22:45:40.520691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.520624 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dde28fb-3925-4b15-9bb9-76c654c5b88c-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:40.520691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:40.520655 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27j2d\" (UniqueName: \"kubernetes.io/projected/5dde28fb-3925-4b15-9bb9-76c654c5b88c-kube-api-access-27j2d\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:41.117192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:41.116958 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" event={"ID":"5dde28fb-3925-4b15-9bb9-76c654c5b88c","Type":"ContainerDied","Data":"2fbdbff245eca25c9e175c5fc8144f7eb9534a480c21c31278751d5eb6aa5ffd"} Nov 01 22:45:41.117192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:41.117025 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fbdbff245eca25c9e175c5fc8144f7eb9534a480c21c31278751d5eb6aa5ffd" Nov 01 22:45:41.117192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:41.117107 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj" Nov 01 22:45:41.132133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:41.128375 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" event={"ID":"2a4657fe-6a6b-4478-bcee-27b745b82cf1","Type":"ContainerStarted","Data":"eca48feed66cf571235fca83cd3fe383e49fbeecc80e4ff3bd14a812bd925531"} Nov 01 22:45:41.132133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:41.128443 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" event={"ID":"2a4657fe-6a6b-4478-bcee-27b745b82cf1","Type":"ContainerStarted","Data":"45570529d5dfeda166e1135874d0b07e01e8c78f2f9f64f4d8798da813eaaf4a"} Nov 01 22:45:41.157970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:41.150007 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" podStartSLOduration=2.14998834 podStartE2EDuration="2.14998834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:41.146443219 +0000 UTC m=+1353.794942471" watchObservedRunningTime="2026-11-01 22:45:41.14998834 +0000 UTC m=+1353.798487575" Nov 01 22:45:42.143978 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:42.143945 6287 generic.go:356] "Generic (PLEG): container finished" podID="2a4657fe-6a6b-4478-bcee-27b745b82cf1" containerID="eca48feed66cf571235fca83cd3fe383e49fbeecc80e4ff3bd14a812bd925531" exitCode=0 Nov 01 22:45:42.144456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:42.144029 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" event={"ID":"2a4657fe-6a6b-4478-bcee-27b745b82cf1","Type":"ContainerDied","Data":"eca48feed66cf571235fca83cd3fe383e49fbeecc80e4ff3bd14a812bd925531"} Nov 01 22:45:43.332311 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.332286 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:43.440211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.440133 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4657fe-6a6b-4478-bcee-27b745b82cf1-kube-api-access-szmkk\" (UniqueName: \"kubernetes.io/projected/2a4657fe-6a6b-4478-bcee-27b745b82cf1-kube-api-access-szmkk\") pod \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " Nov 01 22:45:43.440361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.440246 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-xks-ca\") pod \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " Nov 01 22:45:43.440466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.440452 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-openshift-ca\") pod \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\" (UID: \"2a4657fe-6a6b-4478-bcee-27b745b82cf1\") " Nov 01 22:45:43.440812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.440785 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-openshift-ca" pod "2a4657fe-6a6b-4478-bcee-27b745b82cf1" (UID: "2a4657fe-6a6b-4478-bcee-27b745b82cf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:43.441126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.441097 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:43.442259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.442234 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4657fe-6a6b-4478-bcee-27b745b82cf1-kube-api-access-szmkk" pod "2a4657fe-6a6b-4478-bcee-27b745b82cf1" (UID: "2a4657fe-6a6b-4478-bcee-27b745b82cf1"). InnerVolumeSpecName "kube-api-access-szmkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:43.442380 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.442274 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-xks-ca" pod "2a4657fe-6a6b-4478-bcee-27b745b82cf1" (UID: "2a4657fe-6a6b-4478-bcee-27b745b82cf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:43.542798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.542770 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szmkk\" (UniqueName: \"kubernetes.io/projected/2a4657fe-6a6b-4478-bcee-27b745b82cf1-kube-api-access-szmkk\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:43.542798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:43.542797 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4657fe-6a6b-4478-bcee-27b745b82cf1-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.163123 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:44.163091 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" Nov 01 22:45:44.163325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:44.163137 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8" event={"ID":"2a4657fe-6a6b-4478-bcee-27b745b82cf1","Type":"ContainerDied","Data":"45570529d5dfeda166e1135874d0b07e01e8c78f2f9f64f4d8798da813eaaf4a"} Nov 01 22:45:44.163325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:44.163215 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45570529d5dfeda166e1135874d0b07e01e8c78f2f9f64f4d8798da813eaaf4a" Nov 01 22:45:45.597777 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:45.597743 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:45:45.755019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:45.754974 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9"] Nov 01 22:45:47.675599 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:47.675387 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh"] Nov 01 22:45:47.779816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:47.779780 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv"] Nov 01 22:45:47.797227 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:47.797189 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b42cc4-mvjtv"] Nov 01 22:45:48.503061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:48.498696 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh"] Nov 01 22:45:48.510146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:48.508933 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9c08e-xcsdh"] Nov 01 22:45:49.510659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:49.510625 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdffff35-fc26-4169-8373-f92a0d756b26" path="/var/lib/kubelet/pods/cdffff35-fc26-4169-8373-f92a0d756b26/volumes" Nov 01 22:45:49.511983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:49.511959 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2dfd10d-dedb-48e2-a725-1f3efd485f46" path="/var/lib/kubelet/pods/d2dfd10d-dedb-48e2-a725-1f3efd485f46/volumes" Nov 01 22:45:50.299137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:50.297085 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95"] Nov 01 22:45:50.759194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:50.758190 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" podUID="bc55462a-a012-4db1-955f-0f253626f5d2" containerName="payload-processing" containerID="cri-o://57ad8e379e926ddcf01524b16f124209c193e4ebf5c018e9669797e8ad599a64" gracePeriod=25 Nov 01 22:45:50.846139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:50.841559 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:51.484273 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:51.479963 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6"] Nov 01 22:45:52.286479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.282490 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-0fd159-ljkc6"] Nov 01 22:45:52.585222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.582178 6287 generic.go:356] "Generic (PLEG): container finished" podID="bc55462a-a012-4db1-955f-0f253626f5d2" containerID="57ad8e379e926ddcf01524b16f124209c193e4ebf5c018e9669797e8ad599a64" exitCode=0 Nov 01 22:45:52.585222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.582609 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" event={"ID":"bc55462a-a012-4db1-955f-0f253626f5d2","Type":"ContainerDied","Data":"57ad8e379e926ddcf01524b16f124209c193e4ebf5c018e9669797e8ad599a64"} Nov 01 22:45:52.585222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.583816 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:45:52.699135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.698567 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc55462a-a012-4db1-955f-0f253626f5d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc55462a-a012-4db1-955f-0f253626f5d2-plugins-config-volume\") pod \"bc55462a-a012-4db1-955f-0f253626f5d2\" (UID: \"bc55462a-a012-4db1-955f-0f253626f5d2\") " Nov 01 22:45:52.699135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.698681 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc55462a-a012-4db1-955f-0f253626f5d2-kube-api-access-rlnnd\" (UniqueName: \"kubernetes.io/projected/bc55462a-a012-4db1-955f-0f253626f5d2-kube-api-access-rlnnd\") pod \"bc55462a-a012-4db1-955f-0f253626f5d2\" (UID: \"bc55462a-a012-4db1-955f-0f253626f5d2\") " Nov 01 22:45:52.736202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.716894 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc55462a-a012-4db1-955f-0f253626f5d2-plugins-config-volume" pod "bc55462a-a012-4db1-955f-0f253626f5d2" (UID: "bc55462a-a012-4db1-955f-0f253626f5d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:52.736202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.721441 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc55462a-a012-4db1-955f-0f253626f5d2-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:52.736202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.734626 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" podUID="b2e3c8dc-335c-4a2b-a934-6ec7bb99d443" containerName="payload-processing" containerID="cri-o://c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab" gracePeriod=25 Nov 01 22:45:52.815413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.815342 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc55462a-a012-4db1-955f-0f253626f5d2-kube-api-access-rlnnd" pod "bc55462a-a012-4db1-955f-0f253626f5d2" (UID: "bc55462a-a012-4db1-955f-0f253626f5d2"). InnerVolumeSpecName "kube-api-access-rlnnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:52.842967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.826511 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlnnd\" (UniqueName: \"kubernetes.io/projected/bc55462a-a012-4db1-955f-0f253626f5d2-kube-api-access-rlnnd\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:52.969869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.969819 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:45:52.970234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:52.970206 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": context deadline exceeded" Nov 01 22:45:53.419799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.419761 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:45:53.477079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.477028 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c9b23d-a634-4275-9bbc-eebe57d2fbd7" path="/var/lib/kubelet/pods/11c9b23d-a634-4275-9bbc-eebe57d2fbd7/volumes" Nov 01 22:45:53.554491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.554460 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-kube-api-access-4h9k5\" (UniqueName: \"kubernetes.io/projected/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-kube-api-access-4h9k5\") pod \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\" (UID: \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\") " Nov 01 22:45:53.554682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.554513 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-plugins-config-volume\") pod \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\" (UID: \"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443\") " Nov 01 22:45:53.554997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.554943 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-plugins-config-volume" pod "b2e3c8dc-335c-4a2b-a934-6ec7bb99d443" (UID: "b2e3c8dc-335c-4a2b-a934-6ec7bb99d443"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:53.556850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.556827 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-kube-api-access-4h9k5" pod "b2e3c8dc-335c-4a2b-a934-6ec7bb99d443" (UID: "b2e3c8dc-335c-4a2b-a934-6ec7bb99d443"). InnerVolumeSpecName "kube-api-access-4h9k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:53.594762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.594729 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" Nov 01 22:45:53.594762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.594742 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9" event={"ID":"bc55462a-a012-4db1-955f-0f253626f5d2","Type":"ContainerDied","Data":"002d96abb7e4c6eeaf4fb27750f5ecd85c2cd6c9bf6ba56322a410c12e031fde"} Nov 01 22:45:53.595051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.594813 6287 scope.go:122] "RemoveContainer" containerID="57ad8e379e926ddcf01524b16f124209c193e4ebf5c018e9669797e8ad599a64" Nov 01 22:45:53.596608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.596485 6287 generic.go:356] "Generic (PLEG): container finished" podID="b2e3c8dc-335c-4a2b-a934-6ec7bb99d443" containerID="c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab" exitCode=0 Nov 01 22:45:53.596608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.596537 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" Nov 01 22:45:53.596608 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.596561 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" event={"ID":"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443","Type":"ContainerDied","Data":"c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab"} Nov 01 22:45:53.596808 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.596627 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh" event={"ID":"b2e3c8dc-335c-4a2b-a934-6ec7bb99d443","Type":"ContainerDied","Data":"8b4bc45bf6eaa841cfc39781bd6a1c57794726ce1d469ce734b0e5660241eb46"} Nov 01 22:45:53.607913 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.607890 6287 scope.go:122] "RemoveContainer" containerID="c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab" Nov 01 22:45:53.620377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.620352 6287 scope.go:122] "RemoveContainer" containerID="c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab" Nov 01 22:45:53.620932 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:45:53.620908 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab\": container with ID starting with c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab not found: ID does not exist" containerID="c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab" Nov 01 22:45:53.621051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.620939 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab"} err="failed to get container status \"c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab\": rpc error: code = NotFound desc = could not find container \"c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab\": container with ID starting with c3bf2958b708aacf09ee8bad399b45969a2b7e59cd4ad5c9e4fdbf75c6a37fab not found: ID does not exist" Nov 01 22:45:53.631394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.631368 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9"] Nov 01 22:45:53.638322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.638299 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0fd159-6595c57d59-xcfq9"] Nov 01 22:45:53.656964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.656888 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h9k5\" (UniqueName: \"kubernetes.io/projected/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-kube-api-access-4h9k5\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:53.656964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.656925 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:53.657891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.657867 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh"] Nov 01 22:45:53.668235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:53.668209 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f2555-cbb67bdb7-txdhh"] Nov 01 22:45:55.298083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.298046 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" podUID="b79b3c37-2479-459b-8c93-3b00a4c9407e" containerName="payload-pre-processing" containerID="cri-o://474426c83eddace490ddf12be2020aa82f036e6594eda22353f5581dd50d2eda" gracePeriod=25 Nov 01 22:45:55.475282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.475245 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e3c8dc-335c-4a2b-a934-6ec7bb99d443" path="/var/lib/kubelet/pods/b2e3c8dc-335c-4a2b-a934-6ec7bb99d443/volumes" Nov 01 22:45:55.475788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.475768 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc55462a-a012-4db1-955f-0f253626f5d2" path="/var/lib/kubelet/pods/bc55462a-a012-4db1-955f-0f253626f5d2/volumes" Nov 01 22:45:55.657352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.654751 6287 generic.go:356] "Generic (PLEG): container finished" podID="b79b3c37-2479-459b-8c93-3b00a4c9407e" containerID="474426c83eddace490ddf12be2020aa82f036e6594eda22353f5581dd50d2eda" exitCode=0 Nov 01 22:45:55.658623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.657777 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" event={"ID":"b79b3c37-2479-459b-8c93-3b00a4c9407e","Type":"ContainerDied","Data":"474426c83eddace490ddf12be2020aa82f036e6594eda22353f5581dd50d2eda"} Nov 01 22:45:55.658623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.658320 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" event={"ID":"b79b3c37-2479-459b-8c93-3b00a4c9407e","Type":"ContainerDied","Data":"7672ab765e70356e185693551f8b1676681c9073540a4e20d64a4fe7846a7b39"} Nov 01 22:45:55.658623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.658369 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7672ab765e70356e185693551f8b1676681c9073540a4e20d64a4fe7846a7b39" Nov 01 22:45:55.679280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.679236 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:45:55.790367 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.790334 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79b3c37-2479-459b-8c93-3b00a4c9407e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b3c37-2479-459b-8c93-3b00a4c9407e-plugins-config-volume\") pod \"b79b3c37-2479-459b-8c93-3b00a4c9407e\" (UID: \"b79b3c37-2479-459b-8c93-3b00a4c9407e\") " Nov 01 22:45:55.790565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.790428 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79b3c37-2479-459b-8c93-3b00a4c9407e-kube-api-access-5qg82\" (UniqueName: \"kubernetes.io/projected/b79b3c37-2479-459b-8c93-3b00a4c9407e-kube-api-access-5qg82\") pod \"b79b3c37-2479-459b-8c93-3b00a4c9407e\" (UID: \"b79b3c37-2479-459b-8c93-3b00a4c9407e\") " Nov 01 22:45:55.790814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.790789 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79b3c37-2479-459b-8c93-3b00a4c9407e-plugins-config-volume" pod "b79b3c37-2479-459b-8c93-3b00a4c9407e" (UID: "b79b3c37-2479-459b-8c93-3b00a4c9407e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:55.791167 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.791096 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79b3c37-2479-459b-8c93-3b00a4c9407e-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:55.793208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.793176 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79b3c37-2479-459b-8c93-3b00a4c9407e-kube-api-access-5qg82" pod "b79b3c37-2479-459b-8c93-3b00a4c9407e" (UID: "b79b3c37-2479-459b-8c93-3b00a4c9407e"). InnerVolumeSpecName "kube-api-access-5qg82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:55.892903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:55.892829 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qg82\" (UniqueName: \"kubernetes.io/projected/b79b3c37-2479-459b-8c93-3b00a4c9407e-kube-api-access-5qg82\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:56.669513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:56.669480 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95" Nov 01 22:45:56.699804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:56.699774 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95"] Nov 01 22:45:56.709534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:56.709505 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-41b8f5-5cd48bf475-t9h95"] Nov 01 22:45:57.473142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:57.473097 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79b3c37-2479-459b-8c93-3b00a4c9407e" path="/var/lib/kubelet/pods/b79b3c37-2479-459b-8c93-3b00a4c9407e/volumes" Nov 01 22:45:58.631154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:58.631098 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt"] Nov 01 22:45:58.631640 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:58.631358 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" containerID="cri-o://5f28d40342eb553fcfef13e06da41ca2614e548079b8a64c71e067e1f27ee29a" gracePeriod=30 Nov 01 22:45:58.635695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:58.635670 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz"] Nov 01 22:45:58.635901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:58.635874 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" containerID="cri-o://be353f7f6586d91f1e3af2e668877affd3a3c5fb5fb4c5e008af842b9fcfbe8c" gracePeriod=30 Nov 01 22:45:58.667775 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:58.667745 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md"] Nov 01 22:45:58.667922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:58.667904 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" containerID="cri-o://d000f5344a67946cdc3400927645ec601966e832829e963d97b6733f0c851ac1" gracePeriod=30 Nov 01 22:45:59.590350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.590312 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x"] Nov 01 22:45:59.591694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591663 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2a4657fe-6a6b-4478-bcee-27b745b82cf1" containerName="revoke-keys" Nov 01 22:45:59.591694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591696 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4657fe-6a6b-4478-bcee-27b745b82cf1" containerName="revoke-keys" Nov 01 22:45:59.591909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591753 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5dde28fb-3925-4b15-9bb9-76c654c5b88c" containerName="revoke-keys" Nov 01 22:45:59.591909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591761 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dde28fb-3925-4b15-9bb9-76c654c5b88c" containerName="revoke-keys" Nov 01 22:45:59.591909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591839 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b2e3c8dc-335c-4a2b-a934-6ec7bb99d443" containerName="payload-processing" Nov 01 22:45:59.591909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591848 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e3c8dc-335c-4a2b-a934-6ec7bb99d443" containerName="payload-processing" Nov 01 22:45:59.591909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591866 6287 cpu_manager.go:414] "containerMap: removing container" podUID="bc55462a-a012-4db1-955f-0f253626f5d2" containerName="payload-processing" Nov 01 22:45:59.591909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591874 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc55462a-a012-4db1-955f-0f253626f5d2" containerName="payload-processing" Nov 01 22:45:59.591909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591907 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b79b3c37-2479-459b-8c93-3b00a4c9407e" containerName="payload-pre-processing" Nov 01 22:45:59.592466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.591915 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79b3c37-2479-459b-8c93-3b00a4c9407e" containerName="payload-pre-processing" Nov 01 22:45:59.592466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.592209 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e3c8dc-335c-4a2b-a934-6ec7bb99d443" containerName="payload-processing" Nov 01 22:45:59.592466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.592224 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc55462a-a012-4db1-955f-0f253626f5d2" containerName="payload-processing" Nov 01 22:45:59.592466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.592246 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79b3c37-2479-459b-8c93-3b00a4c9407e" containerName="payload-pre-processing" Nov 01 22:45:59.592466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.592273 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4657fe-6a6b-4478-bcee-27b745b82cf1" containerName="revoke-keys" Nov 01 22:45:59.592466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.592296 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dde28fb-3925-4b15-9bb9-76c654c5b88c" containerName="revoke-keys" Nov 01 22:45:59.603381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.603359 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.626169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.626135 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x"] Nov 01 22:45:59.667844 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.667809 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1fddbd-openshift-default-dockercfg-5zxzj\"" Nov 01 22:45:59.766225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.766194 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-data\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.766394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.766240 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-credential-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.766394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.766261 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-podinfo\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.766394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.766375 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.766521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.766416 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-certs\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.766577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.766550 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-envoy\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.766658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.766638 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istiod-ca-cert\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.766751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.766728 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-token\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.766905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.766798 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h44d\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-kube-api-access-2h44d\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.867772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.867701 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h44d\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-kube-api-access-2h44d\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.867910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.867795 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-data\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.867910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.867822 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-credential-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.867910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.867843 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-podinfo\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.867910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.867897 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.868107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.867935 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-certs\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.868107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.867995 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-envoy\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.868107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.868046 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istiod-ca-cert\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.868324 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.868157 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-token\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.868324 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.868243 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-data\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.868488 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.868463 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-certs\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.868582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.868474 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-credential-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.868582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.868549 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-socket\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.868969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.868950 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istiod-ca-cert\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.870057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.870040 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-envoy\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.870445 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.870425 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-podinfo\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.880626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.880604 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-token\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.880743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.880723 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h44d\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-kube-api-access-2h44d\") pod \"e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:45:59.934959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:45:59.934934 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:46:00.225255 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:00.225219 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f44cf5_7696_47e4_a64b_4a2c7dac51c2.slice/crio-3c347d12ffeb4fbf8438176f65631a0dc78bdadc3a53123ef1554f1954890f43 WatchSource:0}: Error finding container 3c347d12ffeb4fbf8438176f65631a0dc78bdadc3a53123ef1554f1954890f43: Status 404 returned error can't find the container with id 3c347d12ffeb4fbf8438176f65631a0dc78bdadc3a53123ef1554f1954890f43 Nov 01 22:46:00.226393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:00.226371 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x"] Nov 01 22:46:00.227953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:00.227920 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:46:00.228029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:00.227986 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:46:00.228029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:00.228017 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 22:46:00.707413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:00.707374 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" event={"ID":"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2","Type":"ContainerStarted","Data":"e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035"} Nov 01 22:46:00.707823 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:00.707435 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" event={"ID":"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2","Type":"ContainerStarted","Data":"3c347d12ffeb4fbf8438176f65631a0dc78bdadc3a53123ef1554f1954890f43"} Nov 01 22:46:00.747067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:00.746986 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" podStartSLOduration=1.746972263 podStartE2EDuration="1.746972263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:00.743318621 +0000 UTC m=+1373.391817869" watchObservedRunningTime="2026-11-01 22:46:00.746972263 +0000 UTC m=+1373.395471499" Nov 01 22:46:00.942290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:00.938473 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:46:01.324761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.324725 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2"] Nov 01 22:46:01.329591 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.329563 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.337409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.337047 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:01.337409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.337141 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:01.337409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.337292 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:46:01.337409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.337363 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:01.370148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.367168 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2"] Nov 01 22:46:01.541384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.533275 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.541384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.533377 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtnww\" (UniqueName: \"kubernetes.io/projected/4834b362-ff0c-4d16-83c1-5b9d0c12c013-kube-api-access-dtnww\") pod \"maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.541384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.533458 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.636628 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.636551 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.636790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.636688 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtnww\" (UniqueName: \"kubernetes.io/projected/4834b362-ff0c-4d16-83c1-5b9d0c12c013-kube-api-access-dtnww\") pod \"maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.636790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.636770 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.637390 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.637364 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.639373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.639350 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.668243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.668215 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtnww\" (UniqueName: \"kubernetes.io/projected/4834b362-ff0c-4d16-83c1-5b9d0c12c013-kube-api-access-dtnww\") pod \"maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.700128 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.700080 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:01.969192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.965258 6287 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:46:01.969192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:01.965321 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:46:02.175942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.173284 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s"] Nov 01 22:46:02.204676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.204587 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b71acd-hsk5s"] Nov 01 22:46:02.210813 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.210487 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2"] Nov 01 22:46:02.447212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.447175 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8"] Nov 01 22:46:02.451170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.451147 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-41b8f5-vklk8"] Nov 01 22:46:02.914370 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.914278 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" event={"ID":"4834b362-ff0c-4d16-83c1-5b9d0c12c013","Type":"ContainerStarted","Data":"31e9f9b89b502e08685ab55d826c7460a4fad6e044109c2115704f15c057ae77"} Nov 01 22:46:02.925277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.925236 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" event={"ID":"4834b362-ff0c-4d16-83c1-5b9d0c12c013","Type":"ContainerStarted","Data":"4ae868d01e7f1df959af2833dced7530db3dd17319980a97133bf198283d94df"} Nov 01 22:46:02.936654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.936619 6287 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:46:02.936858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.936678 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:46:03.001875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:02.995208 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" podStartSLOduration=1.9951730159999999 podStartE2EDuration="1.995173016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:02.98487183 +0000 UTC m=+1375.633371069" watchObservedRunningTime="2026-11-01 22:46:02.995173016 +0000 UTC m=+1375.643672253" Nov 01 22:46:03.250142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.245702 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg"] Nov 01 22:46:03.307771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.307721 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg"] Nov 01 22:46:03.308157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.308137 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.411456 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.411404 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.411908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.411884 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.412429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.412042 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strqb\" (UniqueName: \"kubernetes.io/projected/0ddfc083-af80-45ec-82e7-299c3e769fa6-kube-api-access-strqb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.483145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.475445 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4657fe-6a6b-4478-bcee-27b745b82cf1" path="/var/lib/kubelet/pods/2a4657fe-6a6b-4478-bcee-27b745b82cf1/volumes" Nov 01 22:46:03.483145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.476460 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5ca5c5-1fec-4d1b-9177-d643ad080680" path="/var/lib/kubelet/pods/fe5ca5c5-1fec-4d1b-9177-d643ad080680/volumes" Nov 01 22:46:03.513566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.513494 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strqb\" (UniqueName: \"kubernetes.io/projected/0ddfc083-af80-45ec-82e7-299c3e769fa6-kube-api-access-strqb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.513730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.513613 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.513730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.513720 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.514341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.514318 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.516049 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.516025 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.526666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.526642 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-strqb\" (UniqueName: \"kubernetes.io/projected/0ddfc083-af80-45ec-82e7-299c3e769fa6-kube-api-access-strqb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.725814 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.725779 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:03.933297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.930735 6287 generic.go:356] "Generic (PLEG): container finished" podID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerID="be353f7f6586d91f1e3af2e668877affd3a3c5fb5fb4c5e008af842b9fcfbe8c" exitCode=0 Nov 01 22:46:03.933297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.930880 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" event={"ID":"6d0f5f78-2e7f-4b2b-92a1-c893385f265b","Type":"ContainerDied","Data":"be353f7f6586d91f1e3af2e668877affd3a3c5fb5fb4c5e008af842b9fcfbe8c"} Nov 01 22:46:03.933297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.933019 6287 generic.go:356] "Generic (PLEG): container finished" podID="4834b362-ff0c-4d16-83c1-5b9d0c12c013" containerID="31e9f9b89b502e08685ab55d826c7460a4fad6e044109c2115704f15c057ae77" exitCode=0 Nov 01 22:46:03.933297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.933139 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" event={"ID":"4834b362-ff0c-4d16-83c1-5b9d0c12c013","Type":"ContainerDied","Data":"31e9f9b89b502e08685ab55d826c7460a4fad6e044109c2115704f15c057ae77"} Nov 01 22:46:03.935911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.935875 6287 generic.go:356] "Generic (PLEG): container finished" podID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerID="5f28d40342eb553fcfef13e06da41ca2614e548079b8a64c71e067e1f27ee29a" exitCode=0 Nov 01 22:46:03.936062 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.935974 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" event={"ID":"980ff60d-a1fc-4f07-82ff-f916b4686bc5","Type":"ContainerDied","Data":"5f28d40342eb553fcfef13e06da41ca2614e548079b8a64c71e067e1f27ee29a"} Nov 01 22:46:03.937777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.937752 6287 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:46:03.937985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.937962 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:46:03.938616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.938588 6287 generic.go:356] "Generic (PLEG): container finished" podID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerID="d000f5344a67946cdc3400927645ec601966e832829e963d97b6733f0c851ac1" exitCode=0 Nov 01 22:46:03.938717 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:03.938637 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" event={"ID":"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae","Type":"ContainerDied","Data":"d000f5344a67946cdc3400927645ec601966e832829e963d97b6733f0c851ac1"} Nov 01 22:46:04.112500 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:04.112461 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddfc083_af80_45ec_82e7_299c3e769fa6.slice/crio-d1d7d04a4adea7e2d454b439e3ec897f49777164c37534cad8f0ecb85be7f669 WatchSource:0}: Error finding container d1d7d04a4adea7e2d454b439e3ec897f49777164c37534cad8f0ecb85be7f669: Status 404 returned error can't find the container with id d1d7d04a4adea7e2d454b439e3ec897f49777164c37534cad8f0ecb85be7f669 Nov 01 22:46:04.113306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.113282 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg"] Nov 01 22:46:04.167345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.167321 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:46:04.316837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.316779 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:46:04.320537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.320514 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:46:04.331889 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.331866 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-podinfo\") pod \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " Nov 01 22:46:04.331997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.331975 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-kube-api-access-94nqm\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-kube-api-access-94nqm\") pod \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " Nov 01 22:46:04.332052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332038 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-credential-socket\") pod \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " Nov 01 22:46:04.332149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332056 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-certs\") pod \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " Nov 01 22:46:04.332149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332105 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-token\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-token\") pod \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " Nov 01 22:46:04.332275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332231 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-data\") pod \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " Nov 01 22:46:04.332343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332285 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istiod-ca-cert\") pod \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " Nov 01 22:46:04.332343 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332316 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-socket\") pod \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " Nov 01 22:46:04.332453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332408 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-envoy\") pod \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\" (UID: \"6d0f5f78-2e7f-4b2b-92a1-c893385f265b\") " Nov 01 22:46:04.332513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332465 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-credential-socket" pod "6d0f5f78-2e7f-4b2b-92a1-c893385f265b" (UID: "6d0f5f78-2e7f-4b2b-92a1-c893385f265b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.332571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332517 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-certs" pod "6d0f5f78-2e7f-4b2b-92a1-c893385f265b" (UID: "6d0f5f78-2e7f-4b2b-92a1-c893385f265b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.332626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332568 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-data" pod "6d0f5f78-2e7f-4b2b-92a1-c893385f265b" (UID: "6d0f5f78-2e7f-4b2b-92a1-c893385f265b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.332774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332750 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-socket" pod "6d0f5f78-2e7f-4b2b-92a1-c893385f265b" (UID: "6d0f5f78-2e7f-4b2b-92a1-c893385f265b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.332842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.332806 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istiod-ca-cert" pod "6d0f5f78-2e7f-4b2b-92a1-c893385f265b" (UID: "6d0f5f78-2e7f-4b2b-92a1-c893385f265b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:04.333185 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.333158 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.333321 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.333187 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.333321 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.333203 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.333321 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.333217 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.333321 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.333230 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.334334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.334313 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-podinfo" pod "6d0f5f78-2e7f-4b2b-92a1-c893385f265b" (UID: "6d0f5f78-2e7f-4b2b-92a1-c893385f265b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:04.334714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.334701 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-envoy" pod "6d0f5f78-2e7f-4b2b-92a1-c893385f265b" (UID: "6d0f5f78-2e7f-4b2b-92a1-c893385f265b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.334863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.334845 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-token" pod "6d0f5f78-2e7f-4b2b-92a1-c893385f265b" (UID: "6d0f5f78-2e7f-4b2b-92a1-c893385f265b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.334933 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.334878 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-kube-api-access-94nqm" pod "6d0f5f78-2e7f-4b2b-92a1-c893385f265b" (UID: "6d0f5f78-2e7f-4b2b-92a1-c893385f265b"). InnerVolumeSpecName "kube-api-access-94nqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.433975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.433883 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-credential-socket\") pod \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " Nov 01 22:46:04.434209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434006 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-data\") pod \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " Nov 01 22:46:04.434209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434080 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istiod-ca-cert\") pod \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " Nov 01 22:46:04.434209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434138 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-podinfo\") pod \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " Nov 01 22:46:04.434209 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434179 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-socket\") pod \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " Nov 01 22:46:04.434465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434222 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-data\") pod \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " Nov 01 22:46:04.434465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434255 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-token\") pod \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " Nov 01 22:46:04.434465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434343 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-socket\") pod \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " Nov 01 22:46:04.434465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434371 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-certs\") pod \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " Nov 01 22:46:04.434465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434417 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-podinfo\") pod \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " Nov 01 22:46:04.434753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434468 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-token\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-token\") pod \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " Nov 01 22:46:04.434753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434520 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-envoy\") pod \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " Nov 01 22:46:04.434753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434615 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-kube-api-access-q669t\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-kube-api-access-q669t\") pod \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " Nov 01 22:46:04.434753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434672 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-credential-socket\") pod \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " Nov 01 22:46:04.434753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434707 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-certs\") pod \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " Nov 01 22:46:04.435046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434803 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-envoy\") pod \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " Nov 01 22:46:04.435046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434926 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-kube-api-access-stqkz\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-kube-api-access-stqkz\") pod \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\" (UID: \"980ff60d-a1fc-4f07-82ff-f916b4686bc5\") " Nov 01 22:46:04.435046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.434970 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istiod-ca-cert\") pod \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\" (UID: \"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae\") " Nov 01 22:46:04.436469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.435585 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-certs" pod "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" (UID: "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.436469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.435609 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-certs" pod "980ff60d-a1fc-4f07-82ff-f916b4686bc5" (UID: "980ff60d-a1fc-4f07-82ff-f916b4686bc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.436469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.435766 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-credential-socket" pod "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" (UID: "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.436469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.435850 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-credential-socket" pod "980ff60d-a1fc-4f07-82ff-f916b4686bc5" (UID: "980ff60d-a1fc-4f07-82ff-f916b4686bc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.436469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.435944 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istiod-ca-cert" pod "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" (UID: "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:04.436469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.436079 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-socket" pod "980ff60d-a1fc-4f07-82ff-f916b4686bc5" (UID: "980ff60d-a1fc-4f07-82ff-f916b4686bc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.436469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.436152 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-data" pod "980ff60d-a1fc-4f07-82ff-f916b4686bc5" (UID: "980ff60d-a1fc-4f07-82ff-f916b4686bc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.436469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.436360 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-data" pod "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" (UID: "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.437272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.436635 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istiod-ca-cert" pod "980ff60d-a1fc-4f07-82ff-f916b4686bc5" (UID: "980ff60d-a1fc-4f07-82ff-f916b4686bc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:04.437653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.437626 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-socket" pod "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" (UID: "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.437793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.437698 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94nqm\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-kube-api-access-94nqm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.437793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.437721 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.437793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.437736 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.437793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.437751 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.437793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.437765 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.437793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.437780 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0f5f78-2e7f-4b2b-92a1-c893385f265b-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.440916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.440876 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-podinfo" pod "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" (UID: "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:04.440916 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.440903 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-token" pod "980ff60d-a1fc-4f07-82ff-f916b4686bc5" (UID: "980ff60d-a1fc-4f07-82ff-f916b4686bc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.441099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.440932 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-token" pod "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" (UID: "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.441847 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.441814 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-envoy" pod "980ff60d-a1fc-4f07-82ff-f916b4686bc5" (UID: "980ff60d-a1fc-4f07-82ff-f916b4686bc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.442987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.442957 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-envoy" pod "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" (UID: "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:04.442987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.442960 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-kube-api-access-q669t" pod "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" (UID: "7bfe9406-1e6e-4370-8b03-2b5cbb4793ae"). InnerVolumeSpecName "kube-api-access-q669t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.443767 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.443743 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-kube-api-access-stqkz" pod "980ff60d-a1fc-4f07-82ff-f916b4686bc5" (UID: "980ff60d-a1fc-4f07-82ff-f916b4686bc5"). InnerVolumeSpecName "kube-api-access-stqkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:04.444398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.444376 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-podinfo" pod "980ff60d-a1fc-4f07-82ff-f916b4686bc5" (UID: "980ff60d-a1fc-4f07-82ff-f916b4686bc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:04.538761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538724 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqkz\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-kube-api-access-stqkz\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.538761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538755 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.538761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538765 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538776 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538785 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538796 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538804 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538813 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538823 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538834 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538873 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538883 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538900 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538910 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q669t\" (UniqueName: \"kubernetes.io/projected/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae-kube-api-access-q669t\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538919 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.539127 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.538927 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/980ff60d-a1fc-4f07-82ff-f916b4686bc5-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:04.935654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.935571 6287 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:46:04.935654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.935625 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:46:04.948743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.948712 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" event={"ID":"980ff60d-a1fc-4f07-82ff-f916b4686bc5","Type":"ContainerDied","Data":"62021e6a1d037d23cdde1b81ad6332dec840f5558444e486312fcf38afe9108c"} Nov 01 22:46:04.948934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.948781 6287 scope.go:122] "RemoveContainer" containerID="5f28d40342eb553fcfef13e06da41ca2614e548079b8a64c71e067e1f27ee29a" Nov 01 22:46:04.948934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.948721 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt" Nov 01 22:46:04.950445 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.950422 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" Nov 01 22:46:04.950445 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.950429 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md" event={"ID":"7bfe9406-1e6e-4370-8b03-2b5cbb4793ae","Type":"ContainerDied","Data":"601fe584f7bcfc8e54884769f40b30711cc9b08a70c1db90d347970f1933d62e"} Nov 01 22:46:04.952467 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.952439 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" event={"ID":"0ddfc083-af80-45ec-82e7-299c3e769fa6","Type":"ContainerStarted","Data":"6c88108c4b0cf6aa44c70e08aeedc0388bf182cde25373fd84fada58110d40cb"} Nov 01 22:46:04.952594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.952489 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" event={"ID":"0ddfc083-af80-45ec-82e7-299c3e769fa6","Type":"ContainerStarted","Data":"d1d7d04a4adea7e2d454b439e3ec897f49777164c37534cad8f0ecb85be7f669"} Nov 01 22:46:04.954035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.954011 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" Nov 01 22:46:04.954177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.954047 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz" event={"ID":"6d0f5f78-2e7f-4b2b-92a1-c893385f265b","Type":"ContainerDied","Data":"a3c169d838247125f06dd8479a190468a239baadf2386d4be3e253c8d0f08c77"} Nov 01 22:46:04.967850 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:04.967822 6287 scope.go:122] "RemoveContainer" containerID="d000f5344a67946cdc3400927645ec601966e832829e963d97b6733f0c851ac1" Nov 01 22:46:05.010188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.010161 6287 scope.go:122] "RemoveContainer" containerID="be353f7f6586d91f1e3af2e668877affd3a3c5fb5fb4c5e008af842b9fcfbe8c" Nov 01 22:46:05.034159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.034093 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" podStartSLOduration=2.034077973 podStartE2EDuration="2.034077973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:05.000448365 +0000 UTC m=+1377.648947631" watchObservedRunningTime="2026-11-01 22:46:05.034077973 +0000 UTC m=+1377.682577207" Nov 01 22:46:05.037965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.037939 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt"] Nov 01 22:46:05.048436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.048412 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-hlsjt"] Nov 01 22:46:05.108660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.108625 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md"] Nov 01 22:46:05.115631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.115599 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-dq7md"] Nov 01 22:46:05.207750 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.207675 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz"] Nov 01 22:46:05.213049 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.213023 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b42cc4-openshift-default-8c76c4757-gvldz"] Nov 01 22:46:05.356276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.356250 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:05.404614 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.404582 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4834b362-ff0c-4d16-83c1-5b9d0c12c013-kube-api-access-dtnww\" (UniqueName: \"kubernetes.io/projected/4834b362-ff0c-4d16-83c1-5b9d0c12c013-kube-api-access-dtnww\") pod \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " Nov 01 22:46:05.404816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.404645 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-xks-ca\") pod \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " Nov 01 22:46:05.404904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.404860 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-openshift-ca\") pod \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\" (UID: \"4834b362-ff0c-4d16-83c1-5b9d0c12c013\") " Nov 01 22:46:05.405465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.405430 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-openshift-ca" pod "4834b362-ff0c-4d16-83c1-5b9d0c12c013" (UID: "4834b362-ff0c-4d16-83c1-5b9d0c12c013"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:05.407068 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.407042 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4834b362-ff0c-4d16-83c1-5b9d0c12c013-kube-api-access-dtnww" pod "4834b362-ff0c-4d16-83c1-5b9d0c12c013" (UID: "4834b362-ff0c-4d16-83c1-5b9d0c12c013"). InnerVolumeSpecName "kube-api-access-dtnww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:05.407247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.407227 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-xks-ca" pod "4834b362-ff0c-4d16-83c1-5b9d0c12c013" (UID: "4834b362-ff0c-4d16-83c1-5b9d0c12c013"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:05.499486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.499456 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" path="/var/lib/kubelet/pods/6d0f5f78-2e7f-4b2b-92a1-c893385f265b/volumes" Nov 01 22:46:05.500014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.499998 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" path="/var/lib/kubelet/pods/7bfe9406-1e6e-4370-8b03-2b5cbb4793ae/volumes" Nov 01 22:46:05.500482 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.500469 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" path="/var/lib/kubelet/pods/980ff60d-a1fc-4f07-82ff-f916b4686bc5/volumes" Nov 01 22:46:05.507083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.507054 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:05.507218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.507083 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtnww\" (UniqueName: \"kubernetes.io/projected/4834b362-ff0c-4d16-83c1-5b9d0c12c013-kube-api-access-dtnww\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:05.507218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.507101 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4834b362-ff0c-4d16-83c1-5b9d0c12c013-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:05.936436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.936350 6287 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:46:05.936436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.936407 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:46:05.969995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.969958 6287 generic.go:356] "Generic (PLEG): container finished" podID="0ddfc083-af80-45ec-82e7-299c3e769fa6" containerID="6c88108c4b0cf6aa44c70e08aeedc0388bf182cde25373fd84fada58110d40cb" exitCode=0 Nov 01 22:46:05.982144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.970335 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" event={"ID":"0ddfc083-af80-45ec-82e7-299c3e769fa6","Type":"ContainerDied","Data":"6c88108c4b0cf6aa44c70e08aeedc0388bf182cde25373fd84fada58110d40cb"} Nov 01 22:46:05.998903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.998858 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" Nov 01 22:46:05.999062 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.998855 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2" event={"ID":"4834b362-ff0c-4d16-83c1-5b9d0c12c013","Type":"ContainerDied","Data":"4ae868d01e7f1df959af2833dced7530db3dd17319980a97133bf198283d94df"} Nov 01 22:46:05.999062 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:05.998996 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae868d01e7f1df959af2833dced7530db3dd17319980a97133bf198283d94df" Nov 01 22:46:06.935454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:06.935416 6287 patch_prober.go:29] interesting pod/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Nov 01 22:46:06.935900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:06.935476 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Nov 01 22:46:07.281855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.281824 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:07.353320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.353256 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-xks-ca\") pod \"0ddfc083-af80-45ec-82e7-299c3e769fa6\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " Nov 01 22:46:07.353749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.353393 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-openshift-ca\") pod \"0ddfc083-af80-45ec-82e7-299c3e769fa6\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " Nov 01 22:46:07.353749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.353464 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddfc083-af80-45ec-82e7-299c3e769fa6-kube-api-access-strqb\" (UniqueName: \"kubernetes.io/projected/0ddfc083-af80-45ec-82e7-299c3e769fa6-kube-api-access-strqb\") pod \"0ddfc083-af80-45ec-82e7-299c3e769fa6\" (UID: \"0ddfc083-af80-45ec-82e7-299c3e769fa6\") " Nov 01 22:46:07.354275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.354248 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-openshift-ca" pod "0ddfc083-af80-45ec-82e7-299c3e769fa6" (UID: "0ddfc083-af80-45ec-82e7-299c3e769fa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:07.356607 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.356579 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-xks-ca" pod "0ddfc083-af80-45ec-82e7-299c3e769fa6" (UID: "0ddfc083-af80-45ec-82e7-299c3e769fa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:07.356772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.356747 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddfc083-af80-45ec-82e7-299c3e769fa6-kube-api-access-strqb" pod "0ddfc083-af80-45ec-82e7-299c3e769fa6" (UID: "0ddfc083-af80-45ec-82e7-299c3e769fa6"). InnerVolumeSpecName "kube-api-access-strqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:07.455675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.455633 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:07.455675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.455666 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddfc083-af80-45ec-82e7-299c3e769fa6-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:07.455675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.455680 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-strqb\" (UniqueName: \"kubernetes.io/projected/0ddfc083-af80-45ec-82e7-299c3e769fa6-kube-api-access-strqb\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:07.940514 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.940482 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:46:07.941025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.941003 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:46:07.941872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:07.941852 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:46:08.050336 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:08.050294 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" event={"ID":"0ddfc083-af80-45ec-82e7-299c3e769fa6","Type":"ContainerDied","Data":"d1d7d04a4adea7e2d454b439e3ec897f49777164c37534cad8f0ecb85be7f669"} Nov 01 22:46:08.050615 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:08.050597 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1d7d04a4adea7e2d454b439e3ec897f49777164c37534cad8f0ecb85be7f669" Nov 01 22:46:08.050738 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:08.050314 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg" Nov 01 22:46:10.059645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.059604 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb"] Nov 01 22:46:10.060163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.059980 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" containerID="cri-o://107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720" gracePeriod=30 Nov 01 22:46:10.780684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.780652 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb"] Nov 01 22:46:10.781309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781292 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0ddfc083-af80-45ec-82e7-299c3e769fa6" containerName="revoke-keys" Nov 01 22:46:10.781387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781309 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ddfc083-af80-45ec-82e7-299c3e769fa6" containerName="revoke-keys" Nov 01 22:46:10.781387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781325 6287 cpu_manager.go:414] "containerMap: removing container" podUID="4834b362-ff0c-4d16-83c1-5b9d0c12c013" containerName="revoke-keys" Nov 01 22:46:10.781387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781330 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4834b362-ff0c-4d16-83c1-5b9d0c12c013" containerName="revoke-keys" Nov 01 22:46:10.781387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781353 6287 cpu_manager.go:414] "containerMap: removing container" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" Nov 01 22:46:10.781387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781360 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" Nov 01 22:46:10.781387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781387 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" Nov 01 22:46:10.781590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781392 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" Nov 01 22:46:10.781590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781401 6287 cpu_manager.go:414] "containerMap: removing container" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" Nov 01 22:46:10.781590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781405 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" Nov 01 22:46:10.781590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781523 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bfe9406-1e6e-4370-8b03-2b5cbb4793ae" containerName="istio-proxy" Nov 01 22:46:10.781590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781538 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0f5f78-2e7f-4b2b-92a1-c893385f265b" containerName="istio-proxy" Nov 01 22:46:10.781590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781550 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ddfc083-af80-45ec-82e7-299c3e769fa6" containerName="revoke-keys" Nov 01 22:46:10.781590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781561 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="4834b362-ff0c-4d16-83c1-5b9d0c12c013" containerName="revoke-keys" Nov 01 22:46:10.781590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.781575 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="980ff60d-a1fc-4f07-82ff-f916b4686bc5" containerName="istio-proxy" Nov 01 22:46:10.785546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.785527 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:10.793131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.790745 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:10.793131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.791354 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:46:10.796131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.793415 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:10.796131 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.793698 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:10.796519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.796239 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb"] Nov 01 22:46:10.808267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.808237 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm54n\" (UniqueName: \"kubernetes.io/projected/087e2b20-27e1-4d95-b0e1-d3484b86571b-kube-api-access-fm54n\") pod \"maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:10.808405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.808319 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:10.808405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.808360 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:10.909662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.909629 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm54n\" (UniqueName: \"kubernetes.io/projected/087e2b20-27e1-4d95-b0e1-d3484b86571b-kube-api-access-fm54n\") pod \"maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:10.909855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.909694 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:10.909855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.909732 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:10.910420 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.910390 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:10.912179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.912158 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:10.920804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:10.918818 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm54n\" (UniqueName: \"kubernetes.io/projected/087e2b20-27e1-4d95-b0e1-d3484b86571b-kube-api-access-fm54n\") pod \"maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:11.104513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:11.104433 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:11.391319 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:11.391278 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087e2b20_27e1_4d95_b0e1_d3484b86571b.slice/crio-dfa3598ea9498706847422255d01d101bc6507f0da9d3692ae3c98b389b4b7d1 WatchSource:0}: Error finding container dfa3598ea9498706847422255d01d101bc6507f0da9d3692ae3c98b389b4b7d1: Status 404 returned error can't find the container with id dfa3598ea9498706847422255d01d101bc6507f0da9d3692ae3c98b389b4b7d1 Nov 01 22:46:11.391444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:11.391415 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb"] Nov 01 22:46:11.484074 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:11.484038 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm"] Nov 01 22:46:11.678474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:11.678443 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:46:11.678618 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:11.678501 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:46:12.091459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:12.091422 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" event={"ID":"087e2b20-27e1-4d95-b0e1-d3484b86571b","Type":"ContainerStarted","Data":"3591442eb1fdc80d1f3f5a18c4812721c3921ed03de42100cefcb908eac28774"} Nov 01 22:46:12.091628 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:12.091504 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" event={"ID":"087e2b20-27e1-4d95-b0e1-d3484b86571b","Type":"ContainerStarted","Data":"dfa3598ea9498706847422255d01d101bc6507f0da9d3692ae3c98b389b4b7d1"} Nov 01 22:46:12.124680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:12.124622 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" podStartSLOduration=2.124608028 podStartE2EDuration="2.124608028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:12.123369282 +0000 UTC m=+1384.771868516" watchObservedRunningTime="2026-11-01 22:46:12.124608028 +0000 UTC m=+1384.773107263" Nov 01 22:46:13.105102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:13.105059 6287 generic.go:356] "Generic (PLEG): container finished" podID="087e2b20-27e1-4d95-b0e1-d3484b86571b" containerID="3591442eb1fdc80d1f3f5a18c4812721c3921ed03de42100cefcb908eac28774" exitCode=0 Nov 01 22:46:13.105443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:13.105406 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" event={"ID":"087e2b20-27e1-4d95-b0e1-d3484b86571b","Type":"ContainerDied","Data":"3591442eb1fdc80d1f3f5a18c4812721c3921ed03de42100cefcb908eac28774"} Nov 01 22:46:13.429633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:13.429559 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj"] Nov 01 22:46:13.437592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:13.437565 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f2555-q74fj"] Nov 01 22:46:13.473743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:13.473713 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dde28fb-3925-4b15-9bb9-76c654c5b88c" path="/var/lib/kubelet/pods/5dde28fb-3925-4b15-9bb9-76c654c5b88c/volumes" Nov 01 22:46:14.339835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.339809 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:14.404559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.404492 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-xks-ca\") pod \"087e2b20-27e1-4d95-b0e1-d3484b86571b\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " Nov 01 22:46:14.404691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.404650 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-openshift-ca\") pod \"087e2b20-27e1-4d95-b0e1-d3484b86571b\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " Nov 01 22:46:14.404766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.404750 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087e2b20-27e1-4d95-b0e1-d3484b86571b-kube-api-access-fm54n\" (UniqueName: \"kubernetes.io/projected/087e2b20-27e1-4d95-b0e1-d3484b86571b-kube-api-access-fm54n\") pod \"087e2b20-27e1-4d95-b0e1-d3484b86571b\" (UID: \"087e2b20-27e1-4d95-b0e1-d3484b86571b\") " Nov 01 22:46:14.405007 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.404985 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-openshift-ca" pod "087e2b20-27e1-4d95-b0e1-d3484b86571b" (UID: "087e2b20-27e1-4d95-b0e1-d3484b86571b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:14.405307 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.405288 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.406530 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.406511 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-xks-ca" pod "087e2b20-27e1-4d95-b0e1-d3484b86571b" (UID: "087e2b20-27e1-4d95-b0e1-d3484b86571b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:14.406723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.406702 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087e2b20-27e1-4d95-b0e1-d3484b86571b-kube-api-access-fm54n" pod "087e2b20-27e1-4d95-b0e1-d3484b86571b" (UID: "087e2b20-27e1-4d95-b0e1-d3484b86571b"). InnerVolumeSpecName "kube-api-access-fm54n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:14.506760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.506729 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm54n\" (UniqueName: \"kubernetes.io/projected/087e2b20-27e1-4d95-b0e1-d3484b86571b-kube-api-access-fm54n\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.506760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:14.506757 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087e2b20-27e1-4d95-b0e1-d3484b86571b-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.123434 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.123404 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" Nov 01 22:46:15.123550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.123430 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb" event={"ID":"087e2b20-27e1-4d95-b0e1-d3484b86571b","Type":"ContainerDied","Data":"dfa3598ea9498706847422255d01d101bc6507f0da9d3692ae3c98b389b4b7d1"} Nov 01 22:46:15.123550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.123480 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa3598ea9498706847422255d01d101bc6507f0da9d3692ae3c98b389b4b7d1" Nov 01 22:46:15.343364 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.343303 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.418106 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-kube-api-access-z2fxs\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-kube-api-access-z2fxs\") pod \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.418253 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istiod-ca-cert\") pod \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.418293 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-podinfo\") pod \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.418330 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-socket\") pod \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.418357 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-data\") pod \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.418702 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-envoy\") pod \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.418783 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-certs\") pod \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.418926 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-token\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-token\") pod \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.418971 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-credential-socket\") pod \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\" (UID: \"91d2d702-5ea3-46f5-b0ab-fe35808efdc5\") " Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.419649 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-socket" pod "91d2d702-5ea3-46f5-b0ab-fe35808efdc5" (UID: "91d2d702-5ea3-46f5-b0ab-fe35808efdc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.420014 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istiod-ca-cert" pod "91d2d702-5ea3-46f5-b0ab-fe35808efdc5" (UID: "91d2d702-5ea3-46f5-b0ab-fe35808efdc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:15.425138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.421089 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-credential-socket" pod "91d2d702-5ea3-46f5-b0ab-fe35808efdc5" (UID: "91d2d702-5ea3-46f5-b0ab-fe35808efdc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.456203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.425080 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-data" pod "91d2d702-5ea3-46f5-b0ab-fe35808efdc5" (UID: "91d2d702-5ea3-46f5-b0ab-fe35808efdc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.456203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.442464 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-certs" pod "91d2d702-5ea3-46f5-b0ab-fe35808efdc5" (UID: "91d2d702-5ea3-46f5-b0ab-fe35808efdc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.456203 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.448032 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-envoy" pod "91d2d702-5ea3-46f5-b0ab-fe35808efdc5" (UID: "91d2d702-5ea3-46f5-b0ab-fe35808efdc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.458746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.458719 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.458746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.458745 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.458962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.458756 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.458962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.458765 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.458962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.458773 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.458962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.458782 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.459497 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.459462 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-token" pod "91d2d702-5ea3-46f5-b0ab-fe35808efdc5" (UID: "91d2d702-5ea3-46f5-b0ab-fe35808efdc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:15.461327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.461299 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-kube-api-access-z2fxs" pod "91d2d702-5ea3-46f5-b0ab-fe35808efdc5" (UID: "91d2d702-5ea3-46f5-b0ab-fe35808efdc5"). InnerVolumeSpecName "kube-api-access-z2fxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:15.461432 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.461404 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-podinfo" pod "91d2d702-5ea3-46f5-b0ab-fe35808efdc5" (UID: "91d2d702-5ea3-46f5-b0ab-fe35808efdc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:15.561326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.561290 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.561326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.561324 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2fxs\" (UniqueName: \"kubernetes.io/projected/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-kube-api-access-z2fxs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.561782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:15.561339 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91d2d702-5ea3-46f5-b0ab-fe35808efdc5-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.591232 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:15.591196 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:46:16.133444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.133405 6287 generic.go:356] "Generic (PLEG): container finished" podID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerID="107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720" exitCode=0 Nov 01 22:46:16.133642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.133468 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" Nov 01 22:46:16.133642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.133487 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" event={"ID":"91d2d702-5ea3-46f5-b0ab-fe35808efdc5","Type":"ContainerDied","Data":"107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720"} Nov 01 22:46:16.133642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.133549 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb" event={"ID":"91d2d702-5ea3-46f5-b0ab-fe35808efdc5","Type":"ContainerDied","Data":"b2f94b05608a8aa2f76f26ee800ea4ee8d01cc27880ee4762400980ad2e09831"} Nov 01 22:46:16.133642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.133574 6287 scope.go:122] "RemoveContainer" containerID="107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720" Nov 01 22:46:16.149338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.149277 6287 scope.go:122] "RemoveContainer" containerID="62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5" Nov 01 22:46:16.161224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.161191 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb"] Nov 01 22:46:16.166794 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.166767 6287 scope.go:122] "RemoveContainer" containerID="107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720" Nov 01 22:46:16.168934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.168900 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-41b8f5-openshift-default-75fb999d59-r6ldb"] Nov 01 22:46:16.170683 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:16.170651 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720\": container with ID starting with 107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720 not found: ID does not exist" containerID="107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720" Nov 01 22:46:16.170843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.170815 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720"} err="failed to get container status \"107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720\": rpc error: code = NotFound desc = could not find container \"107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720\": container with ID starting with 107d6e6ad3637ee7c822cccbea756d842e32a1904ec3b86e1f08de60a6c9b720 not found: ID does not exist" Nov 01 22:46:16.170947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.170933 6287 scope.go:122] "RemoveContainer" containerID="62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5" Nov 01 22:46:16.171519 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:16.171494 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5\": container with ID starting with 62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5 not found: ID does not exist" containerID="62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5" Nov 01 22:46:16.171597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.171524 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5"} err="failed to get container status \"62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5\": rpc error: code = NotFound desc = could not find container \"62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5\": container with ID starting with 62813220621ccdcd8ef78eac7ca5f0356c0531506ba65e5a02add772057e9bc5 not found: ID does not exist" Nov 01 22:46:16.486183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.484487 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" podUID="7a56c85c-8295-422a-ba5b-6744fa8fae61" containerName="payload-processing" containerID="cri-o://4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f" gracePeriod=25 Nov 01 22:46:16.784154 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.784124 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:46:16.876345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.876306 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a56c85c-8295-422a-ba5b-6744fa8fae61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56c85c-8295-422a-ba5b-6744fa8fae61-plugins-config-volume\") pod \"7a56c85c-8295-422a-ba5b-6744fa8fae61\" (UID: \"7a56c85c-8295-422a-ba5b-6744fa8fae61\") " Nov 01 22:46:16.876549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.876462 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a56c85c-8295-422a-ba5b-6744fa8fae61-kube-api-access-bkjl8\" (UniqueName: \"kubernetes.io/projected/7a56c85c-8295-422a-ba5b-6744fa8fae61-kube-api-access-bkjl8\") pod \"7a56c85c-8295-422a-ba5b-6744fa8fae61\" (UID: \"7a56c85c-8295-422a-ba5b-6744fa8fae61\") " Nov 01 22:46:16.876788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.876755 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a56c85c-8295-422a-ba5b-6744fa8fae61-plugins-config-volume" pod "7a56c85c-8295-422a-ba5b-6744fa8fae61" (UID: "7a56c85c-8295-422a-ba5b-6744fa8fae61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:16.877059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.877035 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a56c85c-8295-422a-ba5b-6744fa8fae61-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:16.878660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.878632 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a56c85c-8295-422a-ba5b-6744fa8fae61-kube-api-access-bkjl8" pod "7a56c85c-8295-422a-ba5b-6744fa8fae61" (UID: "7a56c85c-8295-422a-ba5b-6744fa8fae61"). InnerVolumeSpecName "kube-api-access-bkjl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:16.978498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:16.978459 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkjl8\" (UniqueName: \"kubernetes.io/projected/7a56c85c-8295-422a-ba5b-6744fa8fae61-kube-api-access-bkjl8\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:17.144899 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.144810 6287 generic.go:356] "Generic (PLEG): container finished" podID="7a56c85c-8295-422a-ba5b-6744fa8fae61" containerID="4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f" exitCode=0 Nov 01 22:46:17.144899 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.144857 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" Nov 01 22:46:17.145105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.144902 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" event={"ID":"7a56c85c-8295-422a-ba5b-6744fa8fae61","Type":"ContainerDied","Data":"4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f"} Nov 01 22:46:17.145105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.144978 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm" event={"ID":"7a56c85c-8295-422a-ba5b-6744fa8fae61","Type":"ContainerDied","Data":"042ea8f4831f3fb4e3867f53cb926930f8ab233b357f367942c502613557147f"} Nov 01 22:46:17.145105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.145003 6287 scope.go:122] "RemoveContainer" containerID="4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f" Nov 01 22:46:17.156984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.156966 6287 scope.go:122] "RemoveContainer" containerID="4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f" Nov 01 22:46:17.157335 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:17.157311 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f\": container with ID starting with 4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f not found: ID does not exist" containerID="4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f" Nov 01 22:46:17.157400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.157350 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f"} err="failed to get container status \"4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f\": rpc error: code = NotFound desc = could not find container \"4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f\": container with ID starting with 4a1cae02a32e0a3e57602718bcea341016ee4474380650b841a3efedc16ea54f not found: ID does not exist" Nov 01 22:46:17.178214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.178188 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm"] Nov 01 22:46:17.182042 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.182018 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ae0e3a-65ccb6496c-hf5wm"] Nov 01 22:46:17.472681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.472602 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a56c85c-8295-422a-ba5b-6744fa8fae61" path="/var/lib/kubelet/pods/7a56c85c-8295-422a-ba5b-6744fa8fae61/volumes" Nov 01 22:46:17.473164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:17.473145 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" path="/var/lib/kubelet/pods/91d2d702-5ea3-46f5-b0ab-fe35808efdc5/volumes" Nov 01 22:46:21.200994 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:21.200942 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc"] Nov 01 22:46:21.205950 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:21.204792 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78"] Nov 01 22:46:21.681994 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:21.681955 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l"] Nov 01 22:46:21.682207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:21.682185 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" containerID="cri-o://1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c" gracePeriod=30 Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.261318 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw"] Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.262833 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7a56c85c-8295-422a-ba5b-6744fa8fae61" containerName="payload-processing" Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.262856 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a56c85c-8295-422a-ba5b-6744fa8fae61" containerName="payload-processing" Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.262930 6287 cpu_manager.go:414] "containerMap: removing container" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.262938 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.262963 6287 cpu_manager.go:414] "containerMap: removing container" podUID="087e2b20-27e1-4d95-b0e1-d3484b86571b" containerName="revoke-keys" Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.262970 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087e2b20-27e1-4d95-b0e1-d3484b86571b" containerName="revoke-keys" Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.263177 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a56c85c-8295-422a-ba5b-6744fa8fae61" containerName="payload-processing" Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.263209 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" Nov 01 22:46:22.265138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.263221 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="087e2b20-27e1-4d95-b0e1-d3484b86571b" containerName="revoke-keys" Nov 01 22:46:22.295610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.295577 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.317529 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.317495 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw"] Nov 01 22:46:22.323089 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.323063 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:22.323245 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.323192 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:22.340745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.340680 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:46:22.340853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.340762 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:22.404083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.404040 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aef418-a004-4f24-bff1-034b76521747-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.404272 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.404098 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zsdl\" (UniqueName: \"kubernetes.io/projected/30aef418-a004-4f24-bff1-034b76521747-kube-api-access-6zsdl\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.404360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.404264 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aef418-a004-4f24-bff1-034b76521747-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.506532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.506494 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aef418-a004-4f24-bff1-034b76521747-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.506849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.506824 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aef418-a004-4f24-bff1-034b76521747-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.506977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.506866 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zsdl\" (UniqueName: \"kubernetes.io/projected/30aef418-a004-4f24-bff1-034b76521747-kube-api-access-6zsdl\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.507302 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.507281 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aef418-a004-4f24-bff1-034b76521747-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.509123 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.509097 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aef418-a004-4f24-bff1-034b76521747-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.517724 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.517672 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zsdl\" (UniqueName: \"kubernetes.io/projected/30aef418-a004-4f24-bff1-034b76521747-kube-api-access-6zsdl\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.648502 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.648466 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:22.962301 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:22.962234 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw"] Nov 01 22:46:22.963451 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:22.963416 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30aef418_a004_4f24_bff1_034b76521747.slice/crio-b3e6fc8dcdba7bc10b1eeaea6ac5396bc8ab1c1e92ca5ca4b9ccca16eea955e1 WatchSource:0}: Error finding container b3e6fc8dcdba7bc10b1eeaea6ac5396bc8ab1c1e92ca5ca4b9ccca16eea955e1: Status 404 returned error can't find the container with id b3e6fc8dcdba7bc10b1eeaea6ac5396bc8ab1c1e92ca5ca4b9ccca16eea955e1 Nov 01 22:46:23.341990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:23.341949 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" event={"ID":"30aef418-a004-4f24-bff1-034b76521747","Type":"ContainerStarted","Data":"c2ba21b6add30c93bc81b24eade133c650c4a0e591b47b11769d75fb2d2a7cbb"} Nov 01 22:46:23.342395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:23.342019 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" event={"ID":"30aef418-a004-4f24-bff1-034b76521747","Type":"ContainerStarted","Data":"b3e6fc8dcdba7bc10b1eeaea6ac5396bc8ab1c1e92ca5ca4b9ccca16eea955e1"} Nov 01 22:46:23.370146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:23.370060 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" podStartSLOduration=1.37004551 podStartE2EDuration="1.37004551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:23.368157936 +0000 UTC m=+1396.016657171" watchObservedRunningTime="2026-11-01 22:46:23.37004551 +0000 UTC m=+1396.018544744" Nov 01 22:46:24.351137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:24.351090 6287 generic.go:356] "Generic (PLEG): container finished" podID="30aef418-a004-4f24-bff1-034b76521747" containerID="c2ba21b6add30c93bc81b24eade133c650c4a0e591b47b11769d75fb2d2a7cbb" exitCode=0 Nov 01 22:46:24.351531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:24.351164 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" event={"ID":"30aef418-a004-4f24-bff1-034b76521747","Type":"ContainerDied","Data":"c2ba21b6add30c93bc81b24eade133c650c4a0e591b47b11769d75fb2d2a7cbb"} Nov 01 22:46:25.029996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.029963 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw"] Nov 01 22:46:25.038338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.038306 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bdb1e2-5g8nw"] Nov 01 22:46:25.476664 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.476625 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6fa8f34-4795-4188-835a-08d8bf403220" path="/var/lib/kubelet/pods/a6fa8f34-4795-4188-835a-08d8bf403220/volumes" Nov 01 22:46:25.565061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.565030 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:25.654427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.654344 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30aef418-a004-4f24-bff1-034b76521747-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aef418-a004-4f24-bff1-034b76521747-maas-api-openshift-ca\") pod \"30aef418-a004-4f24-bff1-034b76521747\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " Nov 01 22:46:25.654427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.654395 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30aef418-a004-4f24-bff1-034b76521747-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aef418-a004-4f24-bff1-034b76521747-maas-api-xks-ca\") pod \"30aef418-a004-4f24-bff1-034b76521747\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " Nov 01 22:46:25.654713 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.654674 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30aef418-a004-4f24-bff1-034b76521747-maas-api-openshift-ca" pod "30aef418-a004-4f24-bff1-034b76521747" (UID: "30aef418-a004-4f24-bff1-034b76521747"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:25.654786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.654749 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30aef418-a004-4f24-bff1-034b76521747-kube-api-access-6zsdl\" (UniqueName: \"kubernetes.io/projected/30aef418-a004-4f24-bff1-034b76521747-kube-api-access-6zsdl\") pod \"30aef418-a004-4f24-bff1-034b76521747\" (UID: \"30aef418-a004-4f24-bff1-034b76521747\") " Nov 01 22:46:25.655281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.655260 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aef418-a004-4f24-bff1-034b76521747-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:25.656522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.656500 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30aef418-a004-4f24-bff1-034b76521747-maas-api-xks-ca" pod "30aef418-a004-4f24-bff1-034b76521747" (UID: "30aef418-a004-4f24-bff1-034b76521747"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:25.656710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.656692 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30aef418-a004-4f24-bff1-034b76521747-kube-api-access-6zsdl" pod "30aef418-a004-4f24-bff1-034b76521747" (UID: "30aef418-a004-4f24-bff1-034b76521747"). InnerVolumeSpecName "kube-api-access-6zsdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:25.756865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.756836 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zsdl\" (UniqueName: \"kubernetes.io/projected/30aef418-a004-4f24-bff1-034b76521747-kube-api-access-6zsdl\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:25.756865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:25.756864 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aef418-a004-4f24-bff1-034b76521747-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:26.095754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:26.095723 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg"] Nov 01 22:46:26.099511 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:26.099486 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-fa8cda-4xlvg"] Nov 01 22:46:26.208242 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:26.208194 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" podUID="c46a3da2-5f01-4bba-a31a-a3adc458d938" containerName="payload-pre-processing" containerID="cri-o://c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179" gracePeriod=25 Nov 01 22:46:26.209079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:26.208549 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" podUID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" containerName="payload-processing" containerID="cri-o://4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69" gracePeriod=25 Nov 01 22:46:26.372091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:26.372009 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" Nov 01 22:46:26.372263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:26.372008 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw" event={"ID":"30aef418-a004-4f24-bff1-034b76521747","Type":"ContainerDied","Data":"b3e6fc8dcdba7bc10b1eeaea6ac5396bc8ab1c1e92ca5ca4b9ccca16eea955e1"} Nov 01 22:46:26.372263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:26.372150 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3e6fc8dcdba7bc10b1eeaea6ac5396bc8ab1c1e92ca5ca4b9ccca16eea955e1" Nov 01 22:46:26.977064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:26.977036 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:46:27.085849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.085812 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-envoy\") pod \"c18d254f-3b69-454e-a0b3-b8139db8b020\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " Nov 01 22:46:27.086052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.085884 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-podinfo\") pod \"c18d254f-3b69-454e-a0b3-b8139db8b020\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " Nov 01 22:46:27.086052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086037 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-credential-socket\") pod \"c18d254f-3b69-454e-a0b3-b8139db8b020\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " Nov 01 22:46:27.086217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086161 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-socket\") pod \"c18d254f-3b69-454e-a0b3-b8139db8b020\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " Nov 01 22:46:27.086280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086225 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-data\") pod \"c18d254f-3b69-454e-a0b3-b8139db8b020\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " Nov 01 22:46:27.086280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086270 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-token\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-token\") pod \"c18d254f-3b69-454e-a0b3-b8139db8b020\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " Nov 01 22:46:27.086407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086318 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-credential-socket" pod "c18d254f-3b69-454e-a0b3-b8139db8b020" (UID: "c18d254f-3b69-454e-a0b3-b8139db8b020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:27.086407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086327 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18d254f-3b69-454e-a0b3-b8139db8b020-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d254f-3b69-454e-a0b3-b8139db8b020-istiod-ca-cert\") pod \"c18d254f-3b69-454e-a0b3-b8139db8b020\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " Nov 01 22:46:27.086515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086415 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-socket" pod "c18d254f-3b69-454e-a0b3-b8139db8b020" (UID: "c18d254f-3b69-454e-a0b3-b8139db8b020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:27.086515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086417 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-kube-api-access-5tlpd\") pod \"c18d254f-3b69-454e-a0b3-b8139db8b020\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " Nov 01 22:46:27.086515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086492 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-certs\") pod \"c18d254f-3b69-454e-a0b3-b8139db8b020\" (UID: \"c18d254f-3b69-454e-a0b3-b8139db8b020\") " Nov 01 22:46:27.086727 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086700 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18d254f-3b69-454e-a0b3-b8139db8b020-istiod-ca-cert" pod "c18d254f-3b69-454e-a0b3-b8139db8b020" (UID: "c18d254f-3b69-454e-a0b3-b8139db8b020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:27.086937 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.086908 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-data" pod "c18d254f-3b69-454e-a0b3-b8139db8b020" (UID: "c18d254f-3b69-454e-a0b3-b8139db8b020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:27.087092 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.087065 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-certs" pod "c18d254f-3b69-454e-a0b3-b8139db8b020" (UID: "c18d254f-3b69-454e-a0b3-b8139db8b020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:27.087384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.087366 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:27.087476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.087388 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d254f-3b69-454e-a0b3-b8139db8b020-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:27.087476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.087404 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:27.087476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.087419 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:27.087476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.087433 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:27.088444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.088416 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-podinfo" pod "c18d254f-3b69-454e-a0b3-b8139db8b020" (UID: "c18d254f-3b69-454e-a0b3-b8139db8b020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:27.088553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.088527 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-kube-api-access-5tlpd" pod "c18d254f-3b69-454e-a0b3-b8139db8b020" (UID: "c18d254f-3b69-454e-a0b3-b8139db8b020"). InnerVolumeSpecName "kube-api-access-5tlpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:27.088633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.088607 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-envoy" pod "c18d254f-3b69-454e-a0b3-b8139db8b020" (UID: "c18d254f-3b69-454e-a0b3-b8139db8b020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:27.088855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.088837 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-token" pod "c18d254f-3b69-454e-a0b3-b8139db8b020" (UID: "c18d254f-3b69-454e-a0b3-b8139db8b020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:27.189075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.189046 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:27.189075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.189071 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/c18d254f-3b69-454e-a0b3-b8139db8b020-kube-api-access-5tlpd\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:27.189075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.189081 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:27.189310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.189093 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d254f-3b69-454e-a0b3-b8139db8b020-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:27.382492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.382459 6287 generic.go:356] "Generic (PLEG): container finished" podID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerID="1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c" exitCode=0 Nov 01 22:46:27.382704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.382523 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" Nov 01 22:46:27.382704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.382537 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" event={"ID":"c18d254f-3b69-454e-a0b3-b8139db8b020","Type":"ContainerDied","Data":"1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c"} Nov 01 22:46:27.382704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.382593 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l" event={"ID":"c18d254f-3b69-454e-a0b3-b8139db8b020","Type":"ContainerDied","Data":"d3174db833b228a7851f1af875489c1a15ca232aef260f5b3ed133d760e127cc"} Nov 01 22:46:27.382704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.382616 6287 scope.go:122] "RemoveContainer" containerID="1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c" Nov 01 22:46:27.396226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.396205 6287 scope.go:122] "RemoveContainer" containerID="92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec" Nov 01 22:46:27.411329 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.411300 6287 scope.go:122] "RemoveContainer" containerID="1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c" Nov 01 22:46:27.411761 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:27.411725 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c\": container with ID starting with 1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c not found: ID does not exist" containerID="1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c" Nov 01 22:46:27.411893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.411761 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c"} err="failed to get container status \"1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c\": rpc error: code = NotFound desc = could not find container \"1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c\": container with ID starting with 1a5760275dcff8c50d0893f88c65ead1e77b43c60ea8822e37a4ab9310c00b2c not found: ID does not exist" Nov 01 22:46:27.411893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.411785 6287 scope.go:122] "RemoveContainer" containerID="92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec" Nov 01 22:46:27.412193 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:27.412173 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec\": container with ID starting with 92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec not found: ID does not exist" containerID="92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec" Nov 01 22:46:27.412268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.412199 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec"} err="failed to get container status \"92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec\": rpc error: code = NotFound desc = could not find container \"92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec\": container with ID starting with 92a052489050e67be9abb6c91f1597d0d230cf4cb2da202ee4308e0acd1896ec not found: ID does not exist" Nov 01 22:46:27.414451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.414424 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l"] Nov 01 22:46:27.420683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.420660 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8f2555-openshift-default-7d45ff6f97-b274l"] Nov 01 22:46:27.472479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.472452 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ddfc083-af80-45ec-82e7-299c3e769fa6" path="/var/lib/kubelet/pods/0ddfc083-af80-45ec-82e7-299c3e769fa6/volumes" Nov 01 22:46:27.473026 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:27.473012 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" path="/var/lib/kubelet/pods/c18d254f-3b69-454e-a0b3-b8139db8b020/volumes" Nov 01 22:46:28.696418 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.696384 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt"] Nov 01 22:46:28.697027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.697012 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" Nov 01 22:46:28.697072 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.697028 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" Nov 01 22:46:28.697072 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.697055 6287 cpu_manager.go:414] "containerMap: removing container" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" Nov 01 22:46:28.697072 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.697061 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" Nov 01 22:46:28.697218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.697085 6287 cpu_manager.go:414] "containerMap: removing container" podUID="30aef418-a004-4f24-bff1-034b76521747" containerName="revoke-keys" Nov 01 22:46:28.697218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.697091 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30aef418-a004-4f24-bff1-034b76521747" containerName="revoke-keys" Nov 01 22:46:28.697288 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.697279 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" Nov 01 22:46:28.697326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.697294 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d2d702-5ea3-46f5-b0ab-fe35808efdc5" containerName="istio-proxy" Nov 01 22:46:28.697326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.697305 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="30aef418-a004-4f24-bff1-034b76521747" containerName="revoke-keys" Nov 01 22:46:28.713951 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.713922 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t"] Nov 01 22:46:28.714188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.714064 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.716289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.716150 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" Nov 01 22:46:28.716289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.716186 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" Nov 01 22:46:28.717491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.716452 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18d254f-3b69-454e-a0b3-b8139db8b020" containerName="istio-proxy" Nov 01 22:46:28.719795 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.719737 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:46:28.720061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.719885 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:28.720909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.720889 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:28.722188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.722168 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:28.722313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.722294 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt"] Nov 01 22:46:28.722431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.722419 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:28.727378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.727352 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t"] Nov 01 22:46:28.816438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.816409 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb73e4df-zn49t\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:28.816558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.816483 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e1b140d-5pqzt\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.816558 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.816530 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86ss\" (UniqueName: \"kubernetes.io/projected/f595b3e9-8ea8-4f90-b79c-5532828aacd3-kube-api-access-t86ss\") pod \"maas-api-revoke-sub-eb73e4df-zn49t\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:28.816659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.816604 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb73e4df-zn49t\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:28.816659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.816646 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e1b140d-5pqzt\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.816732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.816674 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29mm\" (UniqueName: \"kubernetes.io/projected/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-kube-api-access-l29mm\") pod \"maas-api-revoke-sub-3e1b140d-5pqzt\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.917503 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.917476 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e1b140d-5pqzt\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.917655 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.917522 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86ss\" (UniqueName: \"kubernetes.io/projected/f595b3e9-8ea8-4f90-b79c-5532828aacd3-kube-api-access-t86ss\") pod \"maas-api-revoke-sub-eb73e4df-zn49t\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:28.917655 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.917553 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb73e4df-zn49t\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:28.917655 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.917586 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e1b140d-5pqzt\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.917655 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.917603 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29mm\" (UniqueName: \"kubernetes.io/projected/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-kube-api-access-l29mm\") pod \"maas-api-revoke-sub-3e1b140d-5pqzt\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.917883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.917773 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb73e4df-zn49t\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:28.918404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.918377 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e1b140d-5pqzt\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.918544 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.918519 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb73e4df-zn49t\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:28.919981 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.919955 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb73e4df-zn49t\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:28.920057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.919959 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e1b140d-5pqzt\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.925914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.925892 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29mm\" (UniqueName: \"kubernetes.io/projected/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-kube-api-access-l29mm\") pod \"maas-api-revoke-sub-3e1b140d-5pqzt\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:28.926305 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:28.926282 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86ss\" (UniqueName: \"kubernetes.io/projected/f595b3e9-8ea8-4f90-b79c-5532828aacd3-kube-api-access-t86ss\") pod \"maas-api-revoke-sub-eb73e4df-zn49t\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:29.055137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:29.054921 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:29.056132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:29.055555 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:29.365468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:29.365441 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt"] Nov 01 22:46:29.366174 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:29.366147 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5a4fb8_2c34_4d00_9c14_f94e8547c731.slice/crio-18bd2b8b52947b65bb32569e288a1265f109ce3167b2cf8268e29f91d970b4a3 WatchSource:0}: Error finding container 18bd2b8b52947b65bb32569e288a1265f109ce3167b2cf8268e29f91d970b4a3: Status 404 returned error can't find the container with id 18bd2b8b52947b65bb32569e288a1265f109ce3167b2cf8268e29f91d970b4a3 Nov 01 22:46:29.367694 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:29.367669 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf595b3e9_8ea8_4f90_b79c_5532828aacd3.slice/crio-7edd872450bc738adcf477c834670353ed7b1f6695773103c434bed4e2d266f2 WatchSource:0}: Error finding container 7edd872450bc738adcf477c834670353ed7b1f6695773103c434bed4e2d266f2: Status 404 returned error can't find the container with id 7edd872450bc738adcf477c834670353ed7b1f6695773103c434bed4e2d266f2 Nov 01 22:46:29.369774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:29.369754 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t"] Nov 01 22:46:29.428955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:29.428931 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" event={"ID":"f595b3e9-8ea8-4f90-b79c-5532828aacd3","Type":"ContainerStarted","Data":"7edd872450bc738adcf477c834670353ed7b1f6695773103c434bed4e2d266f2"} Nov 01 22:46:29.430344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:29.430323 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" event={"ID":"2d5a4fb8-2c34-4d00-9c14-f94e8547c731","Type":"ContainerStarted","Data":"18bd2b8b52947b65bb32569e288a1265f109ce3167b2cf8268e29f91d970b4a3"} Nov 01 22:46:30.441092 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:30.441056 6287 generic.go:356] "Generic (PLEG): container finished" podID="f595b3e9-8ea8-4f90-b79c-5532828aacd3" containerID="6de786ff767c59344343ae29e19919cdac49bd343512c12e420bf864d38b813d" exitCode=0 Nov 01 22:46:30.441506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:30.441143 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" event={"ID":"f595b3e9-8ea8-4f90-b79c-5532828aacd3","Type":"ContainerDied","Data":"6de786ff767c59344343ae29e19919cdac49bd343512c12e420bf864d38b813d"} Nov 01 22:46:30.443664 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:30.443637 6287 generic.go:356] "Generic (PLEG): container finished" podID="2d5a4fb8-2c34-4d00-9c14-f94e8547c731" containerID="b76d68d10cc05616b22e285048114bf74c7c533d18920e82ee04e943c90565d2" exitCode=0 Nov 01 22:46:30.443793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:30.443716 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" event={"ID":"2d5a4fb8-2c34-4d00-9c14-f94e8547c731","Type":"ContainerDied","Data":"b76d68d10cc05616b22e285048114bf74c7c533d18920e82ee04e943c90565d2"} Nov 01 22:46:30.584638 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:30.577500 6287 scope.go:122] "RemoveContainer" containerID="2c2f2639ab380b613a70ccf05cb008ab2df31eabbae7467751df4eb5482f4aef" Nov 01 22:46:30.598031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:30.597986 6287 scope.go:122] "RemoveContainer" containerID="474426c83eddace490ddf12be2020aa82f036e6594eda22353f5581dd50d2eda" Nov 01 22:46:31.670954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.670929 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:31.678074 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.678048 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-xks-ca\") pod \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " Nov 01 22:46:31.678237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.678099 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-kube-api-access-l29mm\" (UniqueName: \"kubernetes.io/projected/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-kube-api-access-l29mm\") pod \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " Nov 01 22:46:31.678237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.678202 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-openshift-ca\") pod \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\" (UID: \"2d5a4fb8-2c34-4d00-9c14-f94e8547c731\") " Nov 01 22:46:31.678600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.678576 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-openshift-ca" pod "2d5a4fb8-2c34-4d00-9c14-f94e8547c731" (UID: "2d5a4fb8-2c34-4d00-9c14-f94e8547c731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:31.678843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.678821 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.680304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.680279 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-xks-ca" pod "2d5a4fb8-2c34-4d00-9c14-f94e8547c731" (UID: "2d5a4fb8-2c34-4d00-9c14-f94e8547c731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:31.680410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.680323 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-kube-api-access-l29mm" pod "2d5a4fb8-2c34-4d00-9c14-f94e8547c731" (UID: "2d5a4fb8-2c34-4d00-9c14-f94e8547c731"). InnerVolumeSpecName "kube-api-access-l29mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:31.715688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.715666 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:31.779453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.779422 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f595b3e9-8ea8-4f90-b79c-5532828aacd3-kube-api-access-t86ss\" (UniqueName: \"kubernetes.io/projected/f595b3e9-8ea8-4f90-b79c-5532828aacd3-kube-api-access-t86ss\") pod \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " Nov 01 22:46:31.779653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.779611 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-openshift-ca\") pod \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " Nov 01 22:46:31.779722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.779649 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-xks-ca\") pod \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\" (UID: \"f595b3e9-8ea8-4f90-b79c-5532828aacd3\") " Nov 01 22:46:31.780032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.780000 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-openshift-ca" pod "f595b3e9-8ea8-4f90-b79c-5532828aacd3" (UID: "f595b3e9-8ea8-4f90-b79c-5532828aacd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:31.780306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.780285 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.780366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.780313 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l29mm\" (UniqueName: \"kubernetes.io/projected/2d5a4fb8-2c34-4d00-9c14-f94e8547c731-kube-api-access-l29mm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.780366 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.780329 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.786541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.781898 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f595b3e9-8ea8-4f90-b79c-5532828aacd3-kube-api-access-t86ss" pod "f595b3e9-8ea8-4f90-b79c-5532828aacd3" (UID: "f595b3e9-8ea8-4f90-b79c-5532828aacd3"). InnerVolumeSpecName "kube-api-access-t86ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:31.803062 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.801067 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-xks-ca" pod "f595b3e9-8ea8-4f90-b79c-5532828aacd3" (UID: "f595b3e9-8ea8-4f90-b79c-5532828aacd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:31.881235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.881204 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86ss\" (UniqueName: \"kubernetes.io/projected/f595b3e9-8ea8-4f90-b79c-5532828aacd3-kube-api-access-t86ss\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:31.881235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:31.881231 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f595b3e9-8ea8-4f90-b79c-5532828aacd3-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:32.053944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:32.053855 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" podUID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:46:32.337061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:32.336985 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq"] Nov 01 22:46:32.358931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:32.358890 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng"] Nov 01 22:46:32.494338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:32.494302 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" event={"ID":"2d5a4fb8-2c34-4d00-9c14-f94e8547c731","Type":"ContainerDied","Data":"18bd2b8b52947b65bb32569e288a1265f109ce3167b2cf8268e29f91d970b4a3"} Nov 01 22:46:32.494503 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:32.494358 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18bd2b8b52947b65bb32569e288a1265f109ce3167b2cf8268e29f91d970b4a3" Nov 01 22:46:32.494503 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:32.494314 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt" Nov 01 22:46:32.496002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:32.495976 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" Nov 01 22:46:32.496122 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:32.495999 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t" event={"ID":"f595b3e9-8ea8-4f90-b79c-5532828aacd3","Type":"ContainerDied","Data":"7edd872450bc738adcf477c834670353ed7b1f6695773103c434bed4e2d266f2"} Nov 01 22:46:32.496122 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:32.496041 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7edd872450bc738adcf477c834670353ed7b1f6695773103c434bed4e2d266f2" Nov 01 22:46:33.062598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:33.062555 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" podUID="c46a3da2-5f01-4bba-a31a-a3adc458d938" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:46:34.692297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:34.692265 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2"] Nov 01 22:46:34.704694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:34.704652 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ae0e3a-2ckq2"] Nov 01 22:46:35.437088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:35.437052 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7"] Nov 01 22:46:35.437314 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:35.437287 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" containerID="cri-o://2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895" gracePeriod=30 Nov 01 22:46:35.471736 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:35.471708 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4834b362-ff0c-4d16-83c1-5b9d0c12c013" path="/var/lib/kubelet/pods/4834b362-ff0c-4d16-83c1-5b9d0c12c013/volumes" Nov 01 22:46:36.110188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.110154 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7"] Nov 01 22:46:36.110765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.110748 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f595b3e9-8ea8-4f90-b79c-5532828aacd3" containerName="revoke-keys" Nov 01 22:46:36.110765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.110765 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f595b3e9-8ea8-4f90-b79c-5532828aacd3" containerName="revoke-keys" Nov 01 22:46:36.110899 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.110795 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2d5a4fb8-2c34-4d00-9c14-f94e8547c731" containerName="revoke-keys" Nov 01 22:46:36.110899 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.110800 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5a4fb8-2c34-4d00-9c14-f94e8547c731" containerName="revoke-keys" Nov 01 22:46:36.110966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.110920 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5a4fb8-2c34-4d00-9c14-f94e8547c731" containerName="revoke-keys" Nov 01 22:46:36.110966 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.110941 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f595b3e9-8ea8-4f90-b79c-5532828aacd3" containerName="revoke-keys" Nov 01 22:46:36.115993 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.115974 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.124062 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.123950 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:36.124143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.124066 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:46:36.133820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.133777 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:36.134081 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.134058 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:36.137694 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.137671 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7"] Nov 01 22:46:36.190499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.190462 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.190686 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.190549 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.190772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.190711 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgl6n\" (UniqueName: \"kubernetes.io/projected/6d098ef1-2b4a-4299-a4b9-24d665b34f93-kube-api-access-sgl6n\") pod \"maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.292376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.292334 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgl6n\" (UniqueName: \"kubernetes.io/projected/6d098ef1-2b4a-4299-a4b9-24d665b34f93-kube-api-access-sgl6n\") pod \"maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.292581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.292487 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.292581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.292569 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.293218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.293194 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.294919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.294891 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.301450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.301432 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgl6n\" (UniqueName: \"kubernetes.io/projected/6d098ef1-2b4a-4299-a4b9-24d665b34f93-kube-api-access-sgl6n\") pod \"maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.431194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.431095 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:36.721843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:36.721818 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7"] Nov 01 22:46:36.722608 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:36.722585 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d098ef1_2b4a_4299_a4b9_24d665b34f93.slice/crio-a5eb7ff3135a8d367115a5b966c50c67de805a6b6f0475ed140b309d68db0bf4 WatchSource:0}: Error finding container a5eb7ff3135a8d367115a5b966c50c67de805a6b6f0475ed140b309d68db0bf4: Status 404 returned error can't find the container with id a5eb7ff3135a8d367115a5b966c50c67de805a6b6f0475ed140b309d68db0bf4 Nov 01 22:46:37.337672 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.337631 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" podUID="c6c99002-ccd6-492a-b69a-198d9cb9503f" containerName="payload-processing" containerID="cri-o://0db3909df1d1f285372f7603a69312020fec798b5e9ae3946b5533607ea83d2d" gracePeriod=25 Nov 01 22:46:37.359678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.359494 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerName="payload-pre-processing" containerID="cri-o://acf6580b461d3cb4c18891f2e716bc899d12f6f9f44805fcd9d46785c2a15855" gracePeriod=25 Nov 01 22:46:37.688712 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.688683 6287 generic.go:356] "Generic (PLEG): container finished" podID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerID="acf6580b461d3cb4c18891f2e716bc899d12f6f9f44805fcd9d46785c2a15855" exitCode=0 Nov 01 22:46:37.688908 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.688875 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" event={"ID":"47ee5fcb-2ae6-434f-b22b-03403cd28ba3","Type":"ContainerDied","Data":"acf6580b461d3cb4c18891f2e716bc899d12f6f9f44805fcd9d46785c2a15855"} Nov 01 22:46:37.691872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.691847 6287 generic.go:356] "Generic (PLEG): container finished" podID="6d098ef1-2b4a-4299-a4b9-24d665b34f93" containerID="d51e704c7ef3fa82479bd51e90a504451e29bb80ca66b2bd66ad7136d19314c2" exitCode=0 Nov 01 22:46:37.692012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.691972 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" event={"ID":"6d098ef1-2b4a-4299-a4b9-24d665b34f93","Type":"ContainerDied","Data":"d51e704c7ef3fa82479bd51e90a504451e29bb80ca66b2bd66ad7136d19314c2"} Nov 01 22:46:37.692012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.692002 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" event={"ID":"6d098ef1-2b4a-4299-a4b9-24d665b34f93","Type":"ContainerStarted","Data":"a5eb7ff3135a8d367115a5b966c50c67de805a6b6f0475ed140b309d68db0bf4"} Nov 01 22:46:37.694069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.694045 6287 generic.go:356] "Generic (PLEG): container finished" podID="c6c99002-ccd6-492a-b69a-198d9cb9503f" containerID="0db3909df1d1f285372f7603a69312020fec798b5e9ae3946b5533607ea83d2d" exitCode=0 Nov 01 22:46:37.694210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.694098 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" event={"ID":"c6c99002-ccd6-492a-b69a-198d9cb9503f","Type":"ContainerDied","Data":"0db3909df1d1f285372f7603a69312020fec798b5e9ae3946b5533607ea83d2d"} Nov 01 22:46:37.805510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.805480 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:46:37.838216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.838187 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:46:37.974337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.965624 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-plugins-config-volume\") pod \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\" (UID: \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\") " Nov 01 22:46:37.974337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.965738 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c99002-ccd6-492a-b69a-198d9cb9503f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c99002-ccd6-492a-b69a-198d9cb9503f-plugins-config-volume\") pod \"c6c99002-ccd6-492a-b69a-198d9cb9503f\" (UID: \"c6c99002-ccd6-492a-b69a-198d9cb9503f\") " Nov 01 22:46:37.974337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.965872 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-kube-api-access-wtdnb\" (UniqueName: \"kubernetes.io/projected/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-kube-api-access-wtdnb\") pod \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\" (UID: \"47ee5fcb-2ae6-434f-b22b-03403cd28ba3\") " Nov 01 22:46:37.974337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.965908 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c99002-ccd6-492a-b69a-198d9cb9503f-kube-api-access-zwktx\" (UniqueName: \"kubernetes.io/projected/c6c99002-ccd6-492a-b69a-198d9cb9503f-kube-api-access-zwktx\") pod \"c6c99002-ccd6-492a-b69a-198d9cb9503f\" (UID: \"c6c99002-ccd6-492a-b69a-198d9cb9503f\") " Nov 01 22:46:37.974337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.967735 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-plugins-config-volume" pod "47ee5fcb-2ae6-434f-b22b-03403cd28ba3" (UID: "47ee5fcb-2ae6-434f-b22b-03403cd28ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:37.974337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.968205 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c99002-ccd6-492a-b69a-198d9cb9503f-plugins-config-volume" pod "c6c99002-ccd6-492a-b69a-198d9cb9503f" (UID: "c6c99002-ccd6-492a-b69a-198d9cb9503f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:37.974337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.971643 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-kube-api-access-wtdnb" pod "47ee5fcb-2ae6-434f-b22b-03403cd28ba3" (UID: "47ee5fcb-2ae6-434f-b22b-03403cd28ba3"). InnerVolumeSpecName "kube-api-access-wtdnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:37.974821 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:37.974745 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c99002-ccd6-492a-b69a-198d9cb9503f-kube-api-access-zwktx" pod "c6c99002-ccd6-492a-b69a-198d9cb9503f" (UID: "c6c99002-ccd6-492a-b69a-198d9cb9503f"). InnerVolumeSpecName "kube-api-access-zwktx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:38.068022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.067988 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwktx\" (UniqueName: \"kubernetes.io/projected/c6c99002-ccd6-492a-b69a-198d9cb9503f-kube-api-access-zwktx\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:38.068022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.068020 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:38.068276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.068034 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6c99002-ccd6-492a-b69a-198d9cb9503f-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:38.068276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.068047 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtdnb\" (UniqueName: \"kubernetes.io/projected/47ee5fcb-2ae6-434f-b22b-03403cd28ba3-kube-api-access-wtdnb\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:38.703554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.703471 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" Nov 01 22:46:38.703554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.703508 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng" event={"ID":"47ee5fcb-2ae6-434f-b22b-03403cd28ba3","Type":"ContainerDied","Data":"8abd8ee2ac4392a8beb65689a34c955219aaf05f489b97e3f48572bb7e0aa1f8"} Nov 01 22:46:38.704059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.703567 6287 scope.go:122] "RemoveContainer" containerID="acf6580b461d3cb4c18891f2e716bc899d12f6f9f44805fcd9d46785c2a15855" Nov 01 22:46:38.705422 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.705398 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" Nov 01 22:46:38.705580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.705424 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq" event={"ID":"c6c99002-ccd6-492a-b69a-198d9cb9503f","Type":"ContainerDied","Data":"f6669273328f43cacc287ff47c67fd806f00f7858aa42c7642f1b95cd42bc1bc"} Nov 01 22:46:38.717587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.717568 6287 scope.go:122] "RemoveContainer" containerID="0db3909df1d1f285372f7603a69312020fec798b5e9ae3946b5533607ea83d2d" Nov 01 22:46:38.745763 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.745731 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng"] Nov 01 22:46:38.753218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.753192 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-c9a7ce-ccf774475-n2dng"] Nov 01 22:46:38.774341 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.774310 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq"] Nov 01 22:46:38.783387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.783357 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-c9a7ce-68fc8b95c7-mmtbq"] Nov 01 22:46:38.892368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.892336 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:38.988936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.988904 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d098ef1-2b4a-4299-a4b9-24d665b34f93-kube-api-access-sgl6n\" (UniqueName: \"kubernetes.io/projected/6d098ef1-2b4a-4299-a4b9-24d665b34f93-kube-api-access-sgl6n\") pod \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " Nov 01 22:46:38.989082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.988995 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-openshift-ca\") pod \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " Nov 01 22:46:38.989210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.989108 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-xks-ca\") pod \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\" (UID: \"6d098ef1-2b4a-4299-a4b9-24d665b34f93\") " Nov 01 22:46:38.989431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.989403 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-openshift-ca" pod "6d098ef1-2b4a-4299-a4b9-24d665b34f93" (UID: "6d098ef1-2b4a-4299-a4b9-24d665b34f93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:38.989683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.989660 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:38.991389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.991363 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d098ef1-2b4a-4299-a4b9-24d665b34f93-kube-api-access-sgl6n" pod "6d098ef1-2b4a-4299-a4b9-24d665b34f93" (UID: "6d098ef1-2b4a-4299-a4b9-24d665b34f93"). InnerVolumeSpecName "kube-api-access-sgl6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:38.991593 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:38.991569 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-xks-ca" pod "6d098ef1-2b4a-4299-a4b9-24d665b34f93" (UID: "6d098ef1-2b4a-4299-a4b9-24d665b34f93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:39.091172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:39.091133 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgl6n\" (UniqueName: \"kubernetes.io/projected/6d098ef1-2b4a-4299-a4b9-24d665b34f93-kube-api-access-sgl6n\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:39.091172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:39.091165 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d098ef1-2b4a-4299-a4b9-24d665b34f93-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:39.471658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:39.471580 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" path="/var/lib/kubelet/pods/47ee5fcb-2ae6-434f-b22b-03403cd28ba3/volumes" Nov 01 22:46:39.472031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:39.472017 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c99002-ccd6-492a-b69a-198d9cb9503f" path="/var/lib/kubelet/pods/c6c99002-ccd6-492a-b69a-198d9cb9503f/volumes" Nov 01 22:46:39.715464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:39.715431 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" Nov 01 22:46:39.715464 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:39.715457 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7" event={"ID":"6d098ef1-2b4a-4299-a4b9-24d665b34f93","Type":"ContainerDied","Data":"a5eb7ff3135a8d367115a5b966c50c67de805a6b6f0475ed140b309d68db0bf4"} Nov 01 22:46:39.715980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:39.715508 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5eb7ff3135a8d367115a5b966c50c67de805a6b6f0475ed140b309d68db0bf4" Nov 01 22:46:40.723126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.723024 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:46:40.728935 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.728904 6287 generic.go:356] "Generic (PLEG): container finished" podID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerID="2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895" exitCode=0 Nov 01 22:46:40.729076 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.728967 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" Nov 01 22:46:40.729076 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.728986 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" event={"ID":"184370a9-d837-4d2a-8e83-72e7b3f6ebee","Type":"ContainerDied","Data":"2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895"} Nov 01 22:46:40.729076 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.729053 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7" event={"ID":"184370a9-d837-4d2a-8e83-72e7b3f6ebee","Type":"ContainerDied","Data":"fdc5979da459e58456e9ad981fe87ce84a0d2f4bfac733b93b720cd5cf0395ad"} Nov 01 22:46:40.729840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.729078 6287 scope.go:122] "RemoveContainer" containerID="2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895" Nov 01 22:46:40.741715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.741692 6287 scope.go:122] "RemoveContainer" containerID="4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561" Nov 01 22:46:40.752594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.752571 6287 scope.go:122] "RemoveContainer" containerID="2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895" Nov 01 22:46:40.752833 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:40.752814 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895\": container with ID starting with 2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895 not found: ID does not exist" containerID="2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895" Nov 01 22:46:40.752888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.752843 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895"} err="failed to get container status \"2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895\": rpc error: code = NotFound desc = could not find container \"2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895\": container with ID starting with 2de0975e08188b42629b691eb6c64709878dafab163672a15bde9e4c29fa2895 not found: ID does not exist" Nov 01 22:46:40.752888 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.752859 6287 scope.go:122] "RemoveContainer" containerID="4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561" Nov 01 22:46:40.753075 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:40.753054 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561\": container with ID starting with 4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561 not found: ID does not exist" containerID="4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561" Nov 01 22:46:40.753145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.753085 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561"} err="failed to get container status \"4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561\": rpc error: code = NotFound desc = could not find container \"4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561\": container with ID starting with 4c5ca6830f48e1e7e4992db474175b34dac58c82ae1eac3b3d9954e859f80561 not found: ID does not exist" Nov 01 22:46:40.812632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.812593 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-podinfo\") pod \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " Nov 01 22:46:40.812852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.812670 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-socket\") pod \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " Nov 01 22:46:40.812852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.812702 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-kube-api-access-zd6pg\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-kube-api-access-zd6pg\") pod \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " Nov 01 22:46:40.812852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.812729 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-token\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-token\") pod \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " Nov 01 22:46:40.812852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.812775 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-envoy\") pod \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " Nov 01 22:46:40.813153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.812864 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-credential-socket\") pod \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " Nov 01 22:46:40.813153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.812903 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-certs\") pod \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " Nov 01 22:46:40.813153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.813064 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istiod-ca-cert\") pod \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " Nov 01 22:46:40.813356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.813233 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-data\") pod \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\" (UID: \"184370a9-d837-4d2a-8e83-72e7b3f6ebee\") " Nov 01 22:46:40.813356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.813289 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-certs" pod "184370a9-d837-4d2a-8e83-72e7b3f6ebee" (UID: "184370a9-d837-4d2a-8e83-72e7b3f6ebee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:40.813705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.813673 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-socket" pod "184370a9-d837-4d2a-8e83-72e7b3f6ebee" (UID: "184370a9-d837-4d2a-8e83-72e7b3f6ebee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:40.813841 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.813763 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istiod-ca-cert" pod "184370a9-d837-4d2a-8e83-72e7b3f6ebee" (UID: "184370a9-d837-4d2a-8e83-72e7b3f6ebee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:40.813942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.813920 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-data" pod "184370a9-d837-4d2a-8e83-72e7b3f6ebee" (UID: "184370a9-d837-4d2a-8e83-72e7b3f6ebee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:40.814019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.813966 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.814019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.813986 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.814019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.814000 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.814174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.814125 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-credential-socket" pod "184370a9-d837-4d2a-8e83-72e7b3f6ebee" (UID: "184370a9-d837-4d2a-8e83-72e7b3f6ebee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:40.818643 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.818617 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-token" pod "184370a9-d837-4d2a-8e83-72e7b3f6ebee" (UID: "184370a9-d837-4d2a-8e83-72e7b3f6ebee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:40.818804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.818658 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-kube-api-access-zd6pg" pod "184370a9-d837-4d2a-8e83-72e7b3f6ebee" (UID: "184370a9-d837-4d2a-8e83-72e7b3f6ebee"). InnerVolumeSpecName "kube-api-access-zd6pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:40.819995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.819970 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-podinfo" pod "184370a9-d837-4d2a-8e83-72e7b3f6ebee" (UID: "184370a9-d837-4d2a-8e83-72e7b3f6ebee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:40.820392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.820366 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-envoy" pod "184370a9-d837-4d2a-8e83-72e7b3f6ebee" (UID: "184370a9-d837-4d2a-8e83-72e7b3f6ebee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:40.863661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.863632 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv"] Nov 01 22:46:40.915368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.915327 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.915368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.915367 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.915553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.915378 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd6pg\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-kube-api-access-zd6pg\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.915553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.915387 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.915553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.915395 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.915553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:40.915404 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/184370a9-d837-4d2a-8e83-72e7b3f6ebee-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:41.080344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:41.080271 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7"] Nov 01 22:46:41.087413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:41.087380 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fa8cda-openshift-default-7999dbd7c9-f97z7"] Nov 01 22:46:41.472214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:41.472141 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" path="/var/lib/kubelet/pods/184370a9-d837-4d2a-8e83-72e7b3f6ebee/volumes" Nov 01 22:46:42.046005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:42.045970 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw"] Nov 01 22:46:42.051318 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:42.051291 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-c9a7ce-j4xjw"] Nov 01 22:46:42.053332 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:42.053306 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" podUID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:46:43.061798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:43.061749 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" podUID="c46a3da2-5f01-4bba-a31a-a3adc458d938" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:46:43.474973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:43.474887 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30aef418-a004-4f24-bff1-034b76521747" path="/var/lib/kubelet/pods/30aef418-a004-4f24-bff1-034b76521747/volumes" Nov 01 22:46:44.254147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254097 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8"] Nov 01 22:46:44.254732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254714 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c6c99002-ccd6-492a-b69a-198d9cb9503f" containerName="payload-processing" Nov 01 22:46:44.254780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254734 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c99002-ccd6-492a-b69a-198d9cb9503f" containerName="payload-processing" Nov 01 22:46:44.254780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254746 6287 cpu_manager.go:414] "containerMap: removing container" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" Nov 01 22:46:44.254780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254751 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" Nov 01 22:46:44.254780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254762 6287 cpu_manager.go:414] "containerMap: removing container" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerName="payload-pre-processing" Nov 01 22:46:44.254780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254767 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerName="payload-pre-processing" Nov 01 22:46:44.254942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254803 6287 cpu_manager.go:414] "containerMap: removing container" podUID="6d098ef1-2b4a-4299-a4b9-24d665b34f93" containerName="revoke-keys" Nov 01 22:46:44.254942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254808 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d098ef1-2b4a-4299-a4b9-24d665b34f93" containerName="revoke-keys" Nov 01 22:46:44.254942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254919 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d098ef1-2b4a-4299-a4b9-24d665b34f93" containerName="revoke-keys" Nov 01 22:46:44.254942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254931 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" Nov 01 22:46:44.254942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254940 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c99002-ccd6-492a-b69a-198d9cb9503f" containerName="payload-processing" Nov 01 22:46:44.255097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254947 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" Nov 01 22:46:44.255097 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.254954 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ee5fcb-2ae6-434f-b22b-03403cd28ba3" containerName="payload-pre-processing" Nov 01 22:46:44.259633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.259613 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.265531 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.265466 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:44.266510 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.266486 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:44.269323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.269301 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:44.269437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.269343 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8"] Nov 01 22:46:44.269437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.269390 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:46:44.286068 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.286044 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.286222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.286076 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.286432 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.286407 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbzf\" (UniqueName: \"kubernetes.io/projected/7cfd83d7-80ae-491b-998f-f25199c72bf4-kube-api-access-wtbzf\") pod \"maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.388226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.388191 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbzf\" (UniqueName: \"kubernetes.io/projected/7cfd83d7-80ae-491b-998f-f25199c72bf4-kube-api-access-wtbzf\") pod \"maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.388378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.388263 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.388378 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.388282 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.388959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.388935 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.390623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.390601 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.396400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.396376 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbzf\" (UniqueName: \"kubernetes.io/projected/7cfd83d7-80ae-491b-998f-f25199c72bf4-kube-api-access-wtbzf\") pod \"maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.576892 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.576810 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:44.867935 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:44.867905 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8"] Nov 01 22:46:44.868090 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:44.868019 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfd83d7_80ae_491b_998f_f25199c72bf4.slice/crio-6f8200e93c82561ed935bf1af0f0396de060cf8826396372f1c20cbf86186970 WatchSource:0}: Error finding container 6f8200e93c82561ed935bf1af0f0396de060cf8826396372f1c20cbf86186970: Status 404 returned error can't find the container with id 6f8200e93c82561ed935bf1af0f0396de060cf8826396372f1c20cbf86186970 Nov 01 22:46:45.589752 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:45.589711 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:46:45.929194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:45.924327 6287 generic.go:356] "Generic (PLEG): container finished" podID="7cfd83d7-80ae-491b-998f-f25199c72bf4" containerID="db0b73dfd82e7fe76ee742f6314e14f5c6e56119f1132e700100a6a505138886" exitCode=0 Nov 01 22:46:45.929194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:45.924532 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" event={"ID":"7cfd83d7-80ae-491b-998f-f25199c72bf4","Type":"ContainerDied","Data":"db0b73dfd82e7fe76ee742f6314e14f5c6e56119f1132e700100a6a505138886"} Nov 01 22:46:45.929194 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:45.924582 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" event={"ID":"7cfd83d7-80ae-491b-998f-f25199c72bf4","Type":"ContainerStarted","Data":"6f8200e93c82561ed935bf1af0f0396de060cf8826396372f1c20cbf86186970"} Nov 01 22:46:45.947161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:45.947083 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz"] Nov 01 22:46:45.947393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:45.947320 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" containerID="cri-o://4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3" gracePeriod=30 Nov 01 22:46:46.310601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.310572 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb"] Nov 01 22:46:46.320513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.320481 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-00e49d-948rb"] Nov 01 22:46:46.486819 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.486550 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt"] Nov 01 22:46:46.493327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.493296 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn"] Nov 01 22:46:46.572345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.572317 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:46:46.614563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.614531 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:46:46.652292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.652256 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46a3da2-5f01-4bba-a31a-a3adc458d938-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46a3da2-5f01-4bba-a31a-a3adc458d938-plugins-config-volume\") pod \"c46a3da2-5f01-4bba-a31a-a3adc458d938\" (UID: \"c46a3da2-5f01-4bba-a31a-a3adc458d938\") " Nov 01 22:46:46.652486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.652306 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46a3da2-5f01-4bba-a31a-a3adc458d938-kube-api-access-m2z59\" (UniqueName: \"kubernetes.io/projected/c46a3da2-5f01-4bba-a31a-a3adc458d938-kube-api-access-m2z59\") pod \"c46a3da2-5f01-4bba-a31a-a3adc458d938\" (UID: \"c46a3da2-5f01-4bba-a31a-a3adc458d938\") " Nov 01 22:46:46.652486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.652461 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9fc308-bd50-44de-b033-3e9cb02b11c7-kube-api-access-vfhbv\" (UniqueName: \"kubernetes.io/projected/5d9fc308-bd50-44de-b033-3e9cb02b11c7-kube-api-access-vfhbv\") pod \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\" (UID: \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\") " Nov 01 22:46:46.652589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.652493 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9fc308-bd50-44de-b033-3e9cb02b11c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fc308-bd50-44de-b033-3e9cb02b11c7-plugins-config-volume\") pod \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\" (UID: \"5d9fc308-bd50-44de-b033-3e9cb02b11c7\") " Nov 01 22:46:46.652697 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.652670 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46a3da2-5f01-4bba-a31a-a3adc458d938-plugins-config-volume" pod "c46a3da2-5f01-4bba-a31a-a3adc458d938" (UID: "c46a3da2-5f01-4bba-a31a-a3adc458d938"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:46.653214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.653184 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c46a3da2-5f01-4bba-a31a-a3adc458d938-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:46.653459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.653437 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9fc308-bd50-44de-b033-3e9cb02b11c7-plugins-config-volume" pod "5d9fc308-bd50-44de-b033-3e9cb02b11c7" (UID: "5d9fc308-bd50-44de-b033-3e9cb02b11c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:46.655487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.655461 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46a3da2-5f01-4bba-a31a-a3adc458d938-kube-api-access-m2z59" pod "c46a3da2-5f01-4bba-a31a-a3adc458d938" (UID: "c46a3da2-5f01-4bba-a31a-a3adc458d938"). InnerVolumeSpecName "kube-api-access-m2z59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:46.655958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.655938 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9fc308-bd50-44de-b033-3e9cb02b11c7-kube-api-access-vfhbv" pod "5d9fc308-bd50-44de-b033-3e9cb02b11c7" (UID: "5d9fc308-bd50-44de-b033-3e9cb02b11c7"). InnerVolumeSpecName "kube-api-access-vfhbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:46.754429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.754349 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2z59\" (UniqueName: \"kubernetes.io/projected/c46a3da2-5f01-4bba-a31a-a3adc458d938-kube-api-access-m2z59\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:46.754429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.754381 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfhbv\" (UniqueName: \"kubernetes.io/projected/5d9fc308-bd50-44de-b033-3e9cb02b11c7-kube-api-access-vfhbv\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:46.754429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.754395 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9fc308-bd50-44de-b033-3e9cb02b11c7-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:46.973517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.973478 6287 generic.go:356] "Generic (PLEG): container finished" podID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" containerID="4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69" exitCode=0 Nov 01 22:46:46.973737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.973540 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" Nov 01 22:46:46.973737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.973560 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" event={"ID":"5d9fc308-bd50-44de-b033-3e9cb02b11c7","Type":"ContainerDied","Data":"4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69"} Nov 01 22:46:46.973737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.973630 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc" event={"ID":"5d9fc308-bd50-44de-b033-3e9cb02b11c7","Type":"ContainerDied","Data":"cd1e6ba218f69c6c6f6f693d6292faa01ac73271eb9781a8e6cdcfe53a643f7f"} Nov 01 22:46:46.973737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.973662 6287 scope.go:122] "RemoveContainer" containerID="4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69" Nov 01 22:46:46.979792 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.978056 6287 generic.go:356] "Generic (PLEG): container finished" podID="c46a3da2-5f01-4bba-a31a-a3adc458d938" containerID="c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179" exitCode=0 Nov 01 22:46:46.979792 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.978336 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" Nov 01 22:46:46.979792 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.978657 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" event={"ID":"c46a3da2-5f01-4bba-a31a-a3adc458d938","Type":"ContainerDied","Data":"c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179"} Nov 01 22:46:46.979792 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.978705 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78" event={"ID":"c46a3da2-5f01-4bba-a31a-a3adc458d938","Type":"ContainerDied","Data":"f3e2873948ebe35e1d16f166aef9d1d19cdd44e0d064217ed5e1bc912fdeb7dc"} Nov 01 22:46:46.989321 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.989301 6287 scope.go:122] "RemoveContainer" containerID="4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69" Nov 01 22:46:46.989586 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:46.989566 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69\": container with ID starting with 4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69 not found: ID does not exist" containerID="4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69" Nov 01 22:46:46.989671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.989592 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69"} err="failed to get container status \"4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69\": rpc error: code = NotFound desc = could not find container \"4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69\": container with ID starting with 4155bc5c6730bd7d3c0387f8392a08c4d574f722d52f1d3cbb0e1eb94319ba69 not found: ID does not exist" Nov 01 22:46:46.989671 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:46.989610 6287 scope.go:122] "RemoveContainer" containerID="c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179" Nov 01 22:46:47.001636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.001531 6287 scope.go:122] "RemoveContainer" containerID="c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179" Nov 01 22:46:47.001892 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:47.001869 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179\": container with ID starting with c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179 not found: ID does not exist" containerID="c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179" Nov 01 22:46:47.002020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.001900 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179"} err="failed to get container status \"c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179\": rpc error: code = NotFound desc = could not find container \"c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179\": container with ID starting with c96ad4b548a9716f7cecb72e38a8c20ebbc86a0a0cdb2bce6cc632bfec8ec179 not found: ID does not exist" Nov 01 22:46:47.007239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.007154 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc"] Nov 01 22:46:47.011743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.011718 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-00e49d-5cd859bccc-2rscc"] Nov 01 22:46:47.024564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.024544 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78"] Nov 01 22:46:47.029451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.029428 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-00e49d-56f496f8cf-ttw78"] Nov 01 22:46:47.130295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.130270 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:47.277997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.277924 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-openshift-ca\") pod \"7cfd83d7-80ae-491b-998f-f25199c72bf4\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " Nov 01 22:46:47.278150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.278049 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-xks-ca\") pod \"7cfd83d7-80ae-491b-998f-f25199c72bf4\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " Nov 01 22:46:47.278150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.278082 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfd83d7-80ae-491b-998f-f25199c72bf4-kube-api-access-wtbzf\" (UniqueName: \"kubernetes.io/projected/7cfd83d7-80ae-491b-998f-f25199c72bf4-kube-api-access-wtbzf\") pod \"7cfd83d7-80ae-491b-998f-f25199c72bf4\" (UID: \"7cfd83d7-80ae-491b-998f-f25199c72bf4\") " Nov 01 22:46:47.278325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.278298 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-openshift-ca" pod "7cfd83d7-80ae-491b-998f-f25199c72bf4" (UID: "7cfd83d7-80ae-491b-998f-f25199c72bf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:47.278655 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.278632 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:47.279974 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.279950 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-xks-ca" pod "7cfd83d7-80ae-491b-998f-f25199c72bf4" (UID: "7cfd83d7-80ae-491b-998f-f25199c72bf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:47.280071 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.280032 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfd83d7-80ae-491b-998f-f25199c72bf4-kube-api-access-wtbzf" pod "7cfd83d7-80ae-491b-998f-f25199c72bf4" (UID: "7cfd83d7-80ae-491b-998f-f25199c72bf4"). InnerVolumeSpecName "kube-api-access-wtbzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:47.379964 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.379933 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfd83d7-80ae-491b-998f-f25199c72bf4-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:47.380137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.379961 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbzf\" (UniqueName: \"kubernetes.io/projected/7cfd83d7-80ae-491b-998f-f25199c72bf4-kube-api-access-wtbzf\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:47.474241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.474208 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087e2b20-27e1-4d95-b0e1-d3484b86571b" path="/var/lib/kubelet/pods/087e2b20-27e1-4d95-b0e1-d3484b86571b/volumes" Nov 01 22:46:47.474669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.474652 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" path="/var/lib/kubelet/pods/5d9fc308-bd50-44de-b033-3e9cb02b11c7/volumes" Nov 01 22:46:47.475040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.475026 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46a3da2-5f01-4bba-a31a-a3adc458d938" path="/var/lib/kubelet/pods/c46a3da2-5f01-4bba-a31a-a3adc458d938/volumes" Nov 01 22:46:47.990238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.990209 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" Nov 01 22:46:47.990238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.990217 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8" event={"ID":"7cfd83d7-80ae-491b-998f-f25199c72bf4","Type":"ContainerDied","Data":"6f8200e93c82561ed935bf1af0f0396de060cf8826396372f1c20cbf86186970"} Nov 01 22:46:47.990726 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:47.990269 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f8200e93c82561ed935bf1af0f0396de060cf8826396372f1c20cbf86186970" Nov 01 22:46:49.059077 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:49.059042 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn"] Nov 01 22:46:49.073144 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:49.073100 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6"] Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.311971 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775"] Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.312910 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" containerName="payload-processing" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.312935 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" containerName="payload-processing" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.312984 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c46a3da2-5f01-4bba-a31a-a3adc458d938" containerName="payload-pre-processing" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.312993 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46a3da2-5f01-4bba-a31a-a3adc458d938" containerName="payload-pre-processing" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.313013 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7cfd83d7-80ae-491b-998f-f25199c72bf4" containerName="revoke-keys" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.313021 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfd83d7-80ae-491b-998f-f25199c72bf4" containerName="revoke-keys" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.313051 6287 cpu_manager.go:414] "containerMap: removing container" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.313059 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184370a9-d837-4d2a-8e83-72e7b3f6ebee" containerName="istio-proxy" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.313214 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46a3da2-5f01-4bba-a31a-a3adc458d938" containerName="payload-pre-processing" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.313229 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfd83d7-80ae-491b-998f-f25199c72bf4" containerName="revoke-keys" Nov 01 22:46:51.338192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.313251 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9fc308-bd50-44de-b033-3e9cb02b11c7" containerName="payload-processing" Nov 01 22:46:51.420371 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.420337 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775"] Nov 01 22:46:51.420796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.420678 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.480745 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.480364 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:51.481153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.481127 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:46:51.502139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.487216 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" podUID="b00ff221-8630-4a3a-b9c3-37e90a1394fa" containerName="payload-processing" containerID="cri-o://33a300b560ac797a0c7c18a2c6dbe9e46ed51f480f13f301adfb08e88d537c97" gracePeriod=25 Nov 01 22:46:51.502139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.495149 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" podUID="5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" containerName="payload-pre-processing" containerID="cri-o://124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748" gracePeriod=25 Nov 01 22:46:51.518152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.510183 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:51.518152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.510269 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:51.554922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.553352 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.554922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.553449 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.554922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.553526 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvbk\" (UniqueName: \"kubernetes.io/projected/670e10db-0c78-44ab-bb15-c445aae81921-kube-api-access-7dvbk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.593175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.592322 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:46:51.657715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.657666 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-certs\") pod \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " Nov 01 22:46:51.658044 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.658006 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-podinfo\") pod \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " Nov 01 22:46:51.658247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.658229 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-token\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-token\") pod \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " Nov 01 22:46:51.658413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.658392 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-socket\") pod \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " Nov 01 22:46:51.658600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.658568 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-data\") pod \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " Nov 01 22:46:51.658790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.658758 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-kube-api-access-cfhh5\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-kube-api-access-cfhh5\") pod \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " Nov 01 22:46:51.659061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.659035 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-envoy\") pod \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " Nov 01 22:46:51.659320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.659296 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istiod-ca-cert\") pod \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " Nov 01 22:46:51.660715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.660692 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-credential-socket\") pod \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\" (UID: \"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f\") " Nov 01 22:46:51.663140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.663099 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.665264 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.664799 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.665983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.665918 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvbk\" (UniqueName: \"kubernetes.io/projected/670e10db-0c78-44ab-bb15-c445aae81921-kube-api-access-7dvbk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.667398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.667376 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-credential-socket" pod "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" (UID: "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:51.668138 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.668098 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-certs" pod "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" (UID: "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:51.673281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.672982 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-socket" pod "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" (UID: "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:51.676318 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.674357 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.676542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.676501 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-data" pod "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" (UID: "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:51.677408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.677383 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istiod-ca-cert" pod "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" (UID: "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:51.682776 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.682747 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-envoy" pod "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" (UID: "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:51.683278 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.683240 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-podinfo" pod "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" (UID: "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:51.684733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.684680 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.687796 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.687724 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvbk\" (UniqueName: \"kubernetes.io/projected/670e10db-0c78-44ab-bb15-c445aae81921-kube-api-access-7dvbk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:51.692665 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.692630 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-token" pod "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" (UID: "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:51.694429 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.694402 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-kube-api-access-cfhh5" pod "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" (UID: "c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f"). InnerVolumeSpecName "kube-api-access-cfhh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:51.769309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.769102 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:51.769309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.769177 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:51.769309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.769193 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:51.769309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.769210 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:51.769309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.769225 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:51.769309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.769240 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhh5\" (UniqueName: \"kubernetes.io/projected/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-kube-api-access-cfhh5\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:51.769309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.769256 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:51.769309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.769269 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:51.769309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.769284 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:51.884517 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:51.884418 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:52.387139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.365983 6287 generic.go:356] "Generic (PLEG): container finished" podID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerID="4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3" exitCode=0 Nov 01 22:46:52.387139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.366338 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" event={"ID":"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f","Type":"ContainerDied","Data":"4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3"} Nov 01 22:46:52.387139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.366415 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" event={"ID":"c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f","Type":"ContainerDied","Data":"350d872f694e9bc884963de1785d48c513cbd7fddbe50bc9a5894e8f8c27596f"} Nov 01 22:46:52.387139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.366464 6287 scope.go:122] "RemoveContainer" containerID="4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3" Nov 01 22:46:52.387139 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.366642 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" Nov 01 22:46:52.481147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.480584 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:46:52.481147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.480646 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:46:52.558528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.557756 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz"] Nov 01 22:46:52.558528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.558385 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-t4vrz"] Nov 01 22:46:52.561280 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.561261 6287 scope.go:122] "RemoveContainer" containerID="4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3" Nov 01 22:46:52.561829 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:52.561803 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3\": container with ID starting with 4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3 not found: ID does not exist" containerID="4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3" Nov 01 22:46:52.562001 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.561973 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3"} err="failed to get container status \"4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3\": rpc error: code = NotFound desc = could not find container \"4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3\": container with ID starting with 4c150c8f696ecca3fffe6febb947cf4f24a6ec92c5d2187cf3c3f61a30c168b3 not found: ID does not exist" Nov 01 22:46:52.627231 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:52.627189 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775"] Nov 01 22:46:52.627910 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:52.627881 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670e10db_0c78_44ab_bb15_c445aae81921.slice/crio-b97bee6bb816015c996ac0fc3ba32b14d47e3e970c2d2bdd4274927fe9cae0ec WatchSource:0}: Error finding container b97bee6bb816015c996ac0fc3ba32b14d47e3e970c2d2bdd4274927fe9cae0ec: Status 404 returned error can't find the container with id b97bee6bb816015c996ac0fc3ba32b14d47e3e970c2d2bdd4274927fe9cae0ec Nov 01 22:46:53.383013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:53.382919 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" event={"ID":"670e10db-0c78-44ab-bb15-c445aae81921","Type":"ContainerStarted","Data":"ddf066581c064429aae0f545e13e7d851cdec53f855a6ffbdec8eeff54d76fed"} Nov 01 22:46:53.383013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:53.382996 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" event={"ID":"670e10db-0c78-44ab-bb15-c445aae81921","Type":"ContainerStarted","Data":"b97bee6bb816015c996ac0fc3ba32b14d47e3e970c2d2bdd4274927fe9cae0ec"} Nov 01 22:46:53.407939 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:53.407882 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" podStartSLOduration=2.407869772 podStartE2EDuration="2.407869772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:53.403971098 +0000 UTC m=+1426.052470333" watchObservedRunningTime="2026-11-01 22:46:53.407869772 +0000 UTC m=+1426.056369024" Nov 01 22:46:53.511970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:53.500623 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" path="/var/lib/kubelet/pods/c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f/volumes" Nov 01 22:46:53.541375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:53.541329 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv"] Nov 01 22:46:53.541663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:53.541629 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" containerID="cri-o://b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9" gracePeriod=30 Nov 01 22:46:54.067533 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:54.067489 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" podUID="b00ff221-8630-4a3a-b9c3-37e90a1394fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:46:54.395349 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:54.395261 6287 generic.go:356] "Generic (PLEG): container finished" podID="670e10db-0c78-44ab-bb15-c445aae81921" containerID="ddf066581c064429aae0f545e13e7d851cdec53f855a6ffbdec8eeff54d76fed" exitCode=0 Nov 01 22:46:54.395519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:54.395351 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" event={"ID":"670e10db-0c78-44ab-bb15-c445aae81921","Type":"ContainerDied","Data":"ddf066581c064429aae0f545e13e7d851cdec53f855a6ffbdec8eeff54d76fed"} Nov 01 22:46:54.413661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:54.413612 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs"] Nov 01 22:46:55.066137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.059618 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:46:55.066137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.059677 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:46:55.107137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.105022 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" podUID="5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:46:55.621879 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.621850 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:55.680162 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.680017 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-openshift-ca\") pod \"670e10db-0c78-44ab-bb15-c445aae81921\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " Nov 01 22:46:55.688145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.683334 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-openshift-ca" pod "670e10db-0c78-44ab-bb15-c445aae81921" (UID: "670e10db-0c78-44ab-bb15-c445aae81921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:55.688407 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.688345 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-xks-ca\") pod \"670e10db-0c78-44ab-bb15-c445aae81921\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " Nov 01 22:46:55.688513 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.688463 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670e10db-0c78-44ab-bb15-c445aae81921-kube-api-access-7dvbk\" (UniqueName: \"kubernetes.io/projected/670e10db-0c78-44ab-bb15-c445aae81921-kube-api-access-7dvbk\") pod \"670e10db-0c78-44ab-bb15-c445aae81921\" (UID: \"670e10db-0c78-44ab-bb15-c445aae81921\") " Nov 01 22:46:55.694900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.689158 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:55.719022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.718960 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670e10db-0c78-44ab-bb15-c445aae81921-kube-api-access-7dvbk" pod "670e10db-0c78-44ab-bb15-c445aae81921" (UID: "670e10db-0c78-44ab-bb15-c445aae81921"). InnerVolumeSpecName "kube-api-access-7dvbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:55.748145 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.745251 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-xks-ca" pod "670e10db-0c78-44ab-bb15-c445aae81921" (UID: "670e10db-0c78-44ab-bb15-c445aae81921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:55.815140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.791145 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dvbk\" (UniqueName: \"kubernetes.io/projected/670e10db-0c78-44ab-bb15-c445aae81921-kube-api-access-7dvbk\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:55.816773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.816736 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670e10db-0c78-44ab-bb15-c445aae81921-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:55.901555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:55.899662 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" containerID="cri-o://addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b" gracePeriod=45 Nov 01 22:46:56.319733 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.319694 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:46:56.416482 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.416446 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" Nov 01 22:46:56.416482 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.416461 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775" event={"ID":"670e10db-0c78-44ab-bb15-c445aae81921","Type":"ContainerDied","Data":"b97bee6bb816015c996ac0fc3ba32b14d47e3e970c2d2bdd4274927fe9cae0ec"} Nov 01 22:46:56.416693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.416515 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b97bee6bb816015c996ac0fc3ba32b14d47e3e970c2d2bdd4274927fe9cae0ec" Nov 01 22:46:56.418239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.418215 6287 generic.go:356] "Generic (PLEG): container finished" podID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerID="addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b" exitCode=0 Nov 01 22:46:56.418376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.418265 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" Nov 01 22:46:56.418376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.418299 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" event={"ID":"ca59259c-14aa-4db9-b627-d72c97bbdc61","Type":"ContainerDied","Data":"addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b"} Nov 01 22:46:56.418376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.418343 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv" event={"ID":"ca59259c-14aa-4db9-b627-d72c97bbdc61","Type":"ContainerDied","Data":"05a39dbfc930704f10979d2a6d20d24f20763bcc237a0eeaeccdf5cdd0f1e523"} Nov 01 22:46:56.418376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.418366 6287 scope.go:122] "RemoveContainer" containerID="addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b" Nov 01 22:46:56.435076 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.434662 6287 scope.go:122] "RemoveContainer" containerID="addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b" Nov 01 22:46:56.435241 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:56.435131 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b\": container with ID starting with addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b not found: ID does not exist" containerID="addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b" Nov 01 22:46:56.435241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.435168 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b"} err="failed to get container status \"addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b\": rpc error: code = NotFound desc = could not find container \"addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b\": container with ID starting with addf89104d10b09acb6e08c39e08b06b2a2bc54ea77339051b1419a9ef2c1a7b not found: ID does not exist" Nov 01 22:46:56.449352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.449324 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca59259c-14aa-4db9-b627-d72c97bbdc61-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca59259c-14aa-4db9-b627-d72c97bbdc61-tls-certs\") pod \"ca59259c-14aa-4db9-b627-d72c97bbdc61\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " Nov 01 22:46:56.449458 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.449414 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-tmp-dir\") pod \"ca59259c-14aa-4db9-b627-d72c97bbdc61\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " Nov 01 22:46:56.449548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.449533 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca59259c-14aa-4db9-b627-d72c97bbdc61-kube-api-access-pv2s9\" (UniqueName: \"kubernetes.io/projected/ca59259c-14aa-4db9-b627-d72c97bbdc61-kube-api-access-pv2s9\") pod \"ca59259c-14aa-4db9-b627-d72c97bbdc61\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " Nov 01 22:46:56.449635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.449622 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-dshm\") pod \"ca59259c-14aa-4db9-b627-d72c97bbdc61\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " Nov 01 22:46:56.450592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.449695 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-home\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-home\") pod \"ca59259c-14aa-4db9-b627-d72c97bbdc61\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " Nov 01 22:46:56.450592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.449781 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-model-cache\") pod \"ca59259c-14aa-4db9-b627-d72c97bbdc61\" (UID: \"ca59259c-14aa-4db9-b627-d72c97bbdc61\") " Nov 01 22:46:56.450592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.449826 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-tmp-dir" pod "ca59259c-14aa-4db9-b627-d72c97bbdc61" (UID: "ca59259c-14aa-4db9-b627-d72c97bbdc61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:56.450592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.450345 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-home" pod "ca59259c-14aa-4db9-b627-d72c97bbdc61" (UID: "ca59259c-14aa-4db9-b627-d72c97bbdc61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:56.450592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.450467 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.450592 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.450480 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.451104 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.451064 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-model-cache" pod "ca59259c-14aa-4db9-b627-d72c97bbdc61" (UID: "ca59259c-14aa-4db9-b627-d72c97bbdc61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:56.451912 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.451888 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-dshm" pod "ca59259c-14aa-4db9-b627-d72c97bbdc61" (UID: "ca59259c-14aa-4db9-b627-d72c97bbdc61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:56.452004 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.451906 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca59259c-14aa-4db9-b627-d72c97bbdc61-tls-certs" pod "ca59259c-14aa-4db9-b627-d72c97bbdc61" (UID: "ca59259c-14aa-4db9-b627-d72c97bbdc61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:56.452345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.452326 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca59259c-14aa-4db9-b627-d72c97bbdc61-kube-api-access-pv2s9" pod "ca59259c-14aa-4db9-b627-d72c97bbdc61" (UID: "ca59259c-14aa-4db9-b627-d72c97bbdc61"). InnerVolumeSpecName "kube-api-access-pv2s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:56.554425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.553981 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv2s9\" (UniqueName: \"kubernetes.io/projected/ca59259c-14aa-4db9-b627-d72c97bbdc61-kube-api-access-pv2s9\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.554425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.554251 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.554425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.554278 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca59259c-14aa-4db9-b627-d72c97bbdc61-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.554425 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.554295 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca59259c-14aa-4db9-b627-d72c97bbdc61-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:56.763241 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.763204 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv"] Nov 01 22:46:56.767554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:56.767521 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3d8490/distinct-2-w2-3d8490-kserve-64bf6688d6-xqcnv"] Nov 01 22:46:57.160268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.160228 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz"] Nov 01 22:46:57.161579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.161547 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" Nov 01 22:46:57.161579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.161577 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" Nov 01 22:46:57.161773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.161619 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" Nov 01 22:46:57.161773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.161628 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" Nov 01 22:46:57.161773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.161654 6287 cpu_manager.go:414] "containerMap: removing container" podUID="670e10db-0c78-44ab-bb15-c445aae81921" containerName="revoke-keys" Nov 01 22:46:57.161773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.161663 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670e10db-0c78-44ab-bb15-c445aae81921" containerName="revoke-keys" Nov 01 22:46:57.161961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.161948 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45cef72-e50b-4fe7-9a7c-d4b52a5f0c9f" containerName="istio-proxy" Nov 01 22:46:57.162014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.161966 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" containerName="main" Nov 01 22:46:57.162014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.161982 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="670e10db-0c78-44ab-bb15-c445aae81921" containerName="revoke-keys" Nov 01 22:46:57.169322 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.169289 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx"] Nov 01 22:46:57.172344 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.172317 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.173980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.173959 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz"] Nov 01 22:46:57.174195 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.174038 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.186684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.182086 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx"] Nov 01 22:46:57.186684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.183527 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:57.186684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.183540 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:57.186684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.184798 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:57.186684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.184939 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:46:57.198327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.198283 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98"] Nov 01 22:46:57.202358 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.202339 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.214042 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.214014 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz"] Nov 01 22:46:57.220782 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.220758 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.221049 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.220833 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98"] Nov 01 22:46:57.227628 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.227543 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz"] Nov 01 22:46:57.266579 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.266543 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvz7f\" (UniqueName: \"kubernetes.io/projected/b0ef43af-1174-4cff-8176-72e91a86d6bf-kube-api-access-pvz7f\") pod \"maas-api-revoke-sub-1cd229ef-gg9gx\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.266743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.266620 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd229ef-gg9gx\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.266743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.266685 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwnc\" (UniqueName: \"kubernetes.io/projected/db7cbe18-b450-4519-8ce9-df81d1a3e943-kube-api-access-9jwnc\") pod \"maas-api-revoke-sub-60d96147-98drz\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.266743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.266737 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f6l6\" (UniqueName: \"kubernetes.io/projected/6b1e2363-75e0-4edf-9171-d9166d9a340d-kube-api-access-2f6l6\") pod \"maas-api-revoke-sub-7ed5a44d-c8j98\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.266956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.266919 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd229ef-gg9gx\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.267088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.267064 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e44f43a-lh4hz\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.267193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.267151 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed5a44d-c8j98\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.267268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.267192 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d96147-98drz\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.267268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.267232 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d96147-98drz\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.267356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.267335 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrjq\" (UniqueName: \"kubernetes.io/projected/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-kube-api-access-xkrjq\") pod \"maas-api-revoke-sub-7e44f43a-lh4hz\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.267393 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.267378 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed5a44d-c8j98\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.267445 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.267430 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e44f43a-lh4hz\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.369352 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369314 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvz7f\" (UniqueName: \"kubernetes.io/projected/b0ef43af-1174-4cff-8176-72e91a86d6bf-kube-api-access-pvz7f\") pod \"maas-api-revoke-sub-1cd229ef-gg9gx\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.369566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369371 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd229ef-gg9gx\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.369566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369407 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwnc\" (UniqueName: \"kubernetes.io/projected/db7cbe18-b450-4519-8ce9-df81d1a3e943-kube-api-access-9jwnc\") pod \"maas-api-revoke-sub-60d96147-98drz\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.369566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369438 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f6l6\" (UniqueName: \"kubernetes.io/projected/6b1e2363-75e0-4edf-9171-d9166d9a340d-kube-api-access-2f6l6\") pod \"maas-api-revoke-sub-7ed5a44d-c8j98\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.369566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369466 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd229ef-gg9gx\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.369566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369515 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e44f43a-lh4hz\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.369566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369538 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed5a44d-c8j98\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.369566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369565 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d96147-98drz\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.369955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369603 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d96147-98drz\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.369955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369675 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrjq\" (UniqueName: \"kubernetes.io/projected/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-kube-api-access-xkrjq\") pod \"maas-api-revoke-sub-7e44f43a-lh4hz\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.369955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369705 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed5a44d-c8j98\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.369955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.369758 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e44f43a-lh4hz\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.370319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.370292 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed5a44d-c8j98\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.370402 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.370292 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd229ef-gg9gx\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.370638 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.370538 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e44f43a-lh4hz\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.370726 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.370646 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d96147-98drz\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.372548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.372525 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e44f43a-lh4hz\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.372548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.372539 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed5a44d-c8j98\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.372716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.372625 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd229ef-gg9gx\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.372716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.372650 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d96147-98drz\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.377249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.377227 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwnc\" (UniqueName: \"kubernetes.io/projected/db7cbe18-b450-4519-8ce9-df81d1a3e943-kube-api-access-9jwnc\") pod \"maas-api-revoke-sub-60d96147-98drz\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.377403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.377384 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f6l6\" (UniqueName: \"kubernetes.io/projected/6b1e2363-75e0-4edf-9171-d9166d9a340d-kube-api-access-2f6l6\") pod \"maas-api-revoke-sub-7ed5a44d-c8j98\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.377741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.377725 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvz7f\" (UniqueName: \"kubernetes.io/projected/b0ef43af-1174-4cff-8176-72e91a86d6bf-kube-api-access-pvz7f\") pod \"maas-api-revoke-sub-1cd229ef-gg9gx\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.379035 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.379010 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrjq\" (UniqueName: \"kubernetes.io/projected/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-kube-api-access-xkrjq\") pod \"maas-api-revoke-sub-7e44f43a-lh4hz\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.473395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.473320 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca59259c-14aa-4db9-b627-d72c97bbdc61" path="/var/lib/kubelet/pods/ca59259c-14aa-4db9-b627-d72c97bbdc61/volumes" Nov 01 22:46:57.499641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.499611 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:46:57.507816 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.507792 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:46:57.524379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.524348 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:46:57.543466 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.543432 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:46:57.839244 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:57.839212 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ef43af_1174_4cff_8176_72e91a86d6bf.slice/crio-465149aff52f222e074e732c980a3c2dc55040b158a119b9dae7ec3f7bfce263 WatchSource:0}: Error finding container 465149aff52f222e074e732c980a3c2dc55040b158a119b9dae7ec3f7bfce263: Status 404 returned error can't find the container with id 465149aff52f222e074e732c980a3c2dc55040b158a119b9dae7ec3f7bfce263 Nov 01 22:46:57.841205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.841129 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx"] Nov 01 22:46:57.848649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.848619 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz"] Nov 01 22:46:57.889457 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:57.889429 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1e2363_75e0_4edf_9171_d9166d9a340d.slice/crio-becf5fd4e7e321c508854fbcf42fced4cca60afeace5207f7244703f73452b84 WatchSource:0}: Error finding container becf5fd4e7e321c508854fbcf42fced4cca60afeace5207f7244703f73452b84: Status 404 returned error can't find the container with id becf5fd4e7e321c508854fbcf42fced4cca60afeace5207f7244703f73452b84 Nov 01 22:46:57.890483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.890459 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98"] Nov 01 22:46:57.908148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:57.908122 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz"] Nov 01 22:46:57.909174 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:46:57.909147 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7cbe18_b450_4519_8ce9_df81d1a3e943.slice/crio-903a76f8d306c629de0bb08a2c7d63f4638fce89f481e7365d8cca796849285a WatchSource:0}: Error finding container 903a76f8d306c629de0bb08a2c7d63f4638fce89f481e7365d8cca796849285a: Status 404 returned error can't find the container with id 903a76f8d306c629de0bb08a2c7d63f4638fce89f481e7365d8cca796849285a Nov 01 22:46:58.409285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.408393 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6"] Nov 01 22:46:58.409285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.408476 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5"] Nov 01 22:46:58.409285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.408656 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" containerID="cri-o://1c80a811a55c201d453f3f5b4f349cf9c31ff47adb110838d79102e0879a8c95" gracePeriod=30 Nov 01 22:46:58.409285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.408903 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" containerID="cri-o://c8f0305aa2413a8f06b600611e0f04c0d3a83a7d8eb9e52527d4a6a57b07ba2d" gracePeriod=30 Nov 01 22:46:58.414173 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.414149 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f"] Nov 01 22:46:58.443168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.443138 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" event={"ID":"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1","Type":"ContainerStarted","Data":"4945f548f5e5340df87933c192e1498a2de7b5ecda847e92d2dd348194db5ace"} Nov 01 22:46:58.443330 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.443195 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" event={"ID":"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1","Type":"ContainerStarted","Data":"a18eb78f0452f6136cf24cf66e6df9f5e6d76fe8eb86462fadc5500cfb480d0b"} Nov 01 22:46:58.445539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.445512 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" event={"ID":"6b1e2363-75e0-4edf-9171-d9166d9a340d","Type":"ContainerStarted","Data":"4f927880eb79c6ae35e50475575dd604a3fde5ec59a67585c36b55d94a5083d0"} Nov 01 22:46:58.445662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.445559 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" event={"ID":"6b1e2363-75e0-4edf-9171-d9166d9a340d","Type":"ContainerStarted","Data":"becf5fd4e7e321c508854fbcf42fced4cca60afeace5207f7244703f73452b84"} Nov 01 22:46:58.447668 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.447588 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" event={"ID":"b0ef43af-1174-4cff-8176-72e91a86d6bf","Type":"ContainerStarted","Data":"129a8c1263f4b9b6794d086cbfcd92d88e46c83900a049588cf7d92d55317d0c"} Nov 01 22:46:58.447668 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.447637 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" event={"ID":"b0ef43af-1174-4cff-8176-72e91a86d6bf","Type":"ContainerStarted","Data":"465149aff52f222e074e732c980a3c2dc55040b158a119b9dae7ec3f7bfce263"} Nov 01 22:46:58.449985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.449959 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" event={"ID":"db7cbe18-b450-4519-8ce9-df81d1a3e943","Type":"ContainerStarted","Data":"d4bde02e4c4203ca208ef6cb5b41b532e92e3e25f057a1a913015b18a99633ce"} Nov 01 22:46:58.450102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.450006 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" event={"ID":"db7cbe18-b450-4519-8ce9-df81d1a3e943","Type":"ContainerStarted","Data":"903a76f8d306c629de0bb08a2c7d63f4638fce89f481e7365d8cca796849285a"} Nov 01 22:46:58.450102 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.450086 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" containerID="cri-o://eaa2d2e65024d43527fcbd061f9d567d6dc903304ff41fc3378e7295c6427385" gracePeriod=30 Nov 01 22:46:58.493929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.493881 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" podStartSLOduration=1.493868269 podStartE2EDuration="1.493868269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:58.492063839 +0000 UTC m=+1431.140563073" watchObservedRunningTime="2026-11-01 22:46:58.493868269 +0000 UTC m=+1431.142367503" Nov 01 22:46:58.515224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.515172 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" podStartSLOduration=1.5151543790000002 podStartE2EDuration="1.515154379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:58.51285116 +0000 UTC m=+1431.161350396" watchObservedRunningTime="2026-11-01 22:46:58.515154379 +0000 UTC m=+1431.163653618" Nov 01 22:46:58.553475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.553416 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" podStartSLOduration=1.553398346 podStartE2EDuration="1.553398346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:58.549578957 +0000 UTC m=+1431.198078193" watchObservedRunningTime="2026-11-01 22:46:58.553398346 +0000 UTC m=+1431.201897581" Nov 01 22:46:58.568152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.568087 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" podStartSLOduration=1.568073297 podStartE2EDuration="1.568073297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:58.566008578 +0000 UTC m=+1431.214507836" watchObservedRunningTime="2026-11-01 22:46:58.568073297 +0000 UTC m=+1431.216572531" Nov 01 22:46:58.894059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:58.894033 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:46:59.027163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.026811 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istiod-ca-cert\") pod \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " Nov 01 22:46:59.027163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.026862 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-data\") pod \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " Nov 01 22:46:59.029061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.028911 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-data" pod "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" (UID: "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:59.029061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.026912 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-kube-api-access-sjts7\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-kube-api-access-sjts7\") pod \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " Nov 01 22:46:59.031399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.029654 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-envoy\") pod \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " Nov 01 22:46:59.031399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.029781 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-credential-socket\") pod \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " Nov 01 22:46:59.031399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.029862 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-token\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-token\") pod \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " Nov 01 22:46:59.031399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.030271 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-podinfo\") pod \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " Nov 01 22:46:59.031399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.030354 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-socket\") pod \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " Nov 01 22:46:59.031399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.030464 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-certs\") pod \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\" (UID: \"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83\") " Nov 01 22:46:59.032516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.032234 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-credential-socket" pod "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" (UID: "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:59.032516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.032299 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istiod-ca-cert" pod "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" (UID: "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:59.032516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.032418 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-socket" pod "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" (UID: "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:59.033454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.032896 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-certs" pod "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" (UID: "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:59.033454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.032907 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:59.033454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.032930 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:59.033454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.032946 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:59.033454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.032960 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:59.034410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.034385 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-token" pod "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" (UID: "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:59.034498 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.034403 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-kube-api-access-sjts7" pod "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" (UID: "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83"). InnerVolumeSpecName "kube-api-access-sjts7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:59.035020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.034997 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-envoy" pod "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" (UID: "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:59.035403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.035383 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-podinfo" pod "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" (UID: "bfad33b7-9986-496c-b4cd-b8b8d7a5bf83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:59.134426 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.134395 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjts7\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-kube-api-access-sjts7\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:59.134426 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.134424 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:59.134633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.134434 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:59.134633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.134443 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:59.134633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.134451 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:59.414129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.414077 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" podUID="f58e1b66-6f12-4a7b-93d4-02c43e499f7a" containerName="payload-processing" containerID="cri-o://8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd" gracePeriod=25 Nov 01 22:46:59.460461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.460418 6287 generic.go:356] "Generic (PLEG): container finished" podID="db7cbe18-b450-4519-8ce9-df81d1a3e943" containerID="d4bde02e4c4203ca208ef6cb5b41b532e92e3e25f057a1a913015b18a99633ce" exitCode=0 Nov 01 22:46:59.460658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.460501 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" event={"ID":"db7cbe18-b450-4519-8ce9-df81d1a3e943","Type":"ContainerDied","Data":"d4bde02e4c4203ca208ef6cb5b41b532e92e3e25f057a1a913015b18a99633ce"} Nov 01 22:46:59.462108 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.462073 6287 generic.go:356] "Generic (PLEG): container finished" podID="a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1" containerID="4945f548f5e5340df87933c192e1498a2de7b5ecda847e92d2dd348194db5ace" exitCode=0 Nov 01 22:46:59.462249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.462140 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" event={"ID":"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1","Type":"ContainerDied","Data":"4945f548f5e5340df87933c192e1498a2de7b5ecda847e92d2dd348194db5ace"} Nov 01 22:46:59.463539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.463519 6287 generic.go:356] "Generic (PLEG): container finished" podID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerID="b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9" exitCode=0 Nov 01 22:46:59.463639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.463624 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" Nov 01 22:46:59.465449 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.465426 6287 generic.go:356] "Generic (PLEG): container finished" podID="6b1e2363-75e0-4edf-9171-d9166d9a340d" containerID="4f927880eb79c6ae35e50475575dd604a3fde5ec59a67585c36b55d94a5083d0" exitCode=0 Nov 01 22:46:59.467252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.467228 6287 generic.go:356] "Generic (PLEG): container finished" podID="b0ef43af-1174-4cff-8176-72e91a86d6bf" containerID="129a8c1263f4b9b6794d086cbfcd92d88e46c83900a049588cf7d92d55317d0c" exitCode=0 Nov 01 22:46:59.472152 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.472127 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" event={"ID":"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83","Type":"ContainerDied","Data":"b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9"} Nov 01 22:46:59.472269 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.472177 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv" event={"ID":"bfad33b7-9986-496c-b4cd-b8b8d7a5bf83","Type":"ContainerDied","Data":"55baaee079bc1624111015e8c76f2b78fdd680946fa6cff277dbefc4123dc099"} Nov 01 22:46:59.472333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.472264 6287 scope.go:122] "RemoveContainer" containerID="b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9" Nov 01 22:46:59.472333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.472283 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" event={"ID":"6b1e2363-75e0-4edf-9171-d9166d9a340d","Type":"ContainerDied","Data":"4f927880eb79c6ae35e50475575dd604a3fde5ec59a67585c36b55d94a5083d0"} Nov 01 22:46:59.472333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.472302 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" event={"ID":"b0ef43af-1174-4cff-8176-72e91a86d6bf","Type":"ContainerDied","Data":"129a8c1263f4b9b6794d086cbfcd92d88e46c83900a049588cf7d92d55317d0c"} Nov 01 22:46:59.484555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.484536 6287 scope.go:122] "RemoveContainer" containerID="b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9" Nov 01 22:46:59.484841 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:46:59.484823 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9\": container with ID starting with b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9 not found: ID does not exist" containerID="b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9" Nov 01 22:46:59.484901 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.484849 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9"} err="failed to get container status \"b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9\": rpc error: code = NotFound desc = could not find container \"b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9\": container with ID starting with b78322d23d71365443f18e67dd4fc6b2400b13019f11f551f19c43c5365fe7b9 not found: ID does not exist" Nov 01 22:46:59.498923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.498900 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv"] Nov 01 22:46:59.512295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:46:59.512271 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-9bkgv"] Nov 01 22:47:00.696442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.696363 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:47:00.698376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.698350 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:47:00.779087 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.779054 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:47:00.782474 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.782451 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:47:00.854725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.854698 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-openshift-ca\") pod \"db7cbe18-b450-4519-8ce9-df81d1a3e943\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " Nov 01 22:47:00.854929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.854742 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-openshift-ca\") pod \"b0ef43af-1174-4cff-8176-72e91a86d6bf\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " Nov 01 22:47:00.854929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.854799 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-xks-ca\") pod \"b0ef43af-1174-4cff-8176-72e91a86d6bf\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " Nov 01 22:47:00.854929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.854823 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7cbe18-b450-4519-8ce9-df81d1a3e943-kube-api-access-9jwnc\" (UniqueName: \"kubernetes.io/projected/db7cbe18-b450-4519-8ce9-df81d1a3e943-kube-api-access-9jwnc\") pod \"db7cbe18-b450-4519-8ce9-df81d1a3e943\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " Nov 01 22:47:00.854929 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.854905 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ef43af-1174-4cff-8176-72e91a86d6bf-kube-api-access-pvz7f\" (UniqueName: \"kubernetes.io/projected/b0ef43af-1174-4cff-8176-72e91a86d6bf-kube-api-access-pvz7f\") pod \"b0ef43af-1174-4cff-8176-72e91a86d6bf\" (UID: \"b0ef43af-1174-4cff-8176-72e91a86d6bf\") " Nov 01 22:47:00.855213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.855080 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-xks-ca\") pod \"db7cbe18-b450-4519-8ce9-df81d1a3e943\" (UID: \"db7cbe18-b450-4519-8ce9-df81d1a3e943\") " Nov 01 22:47:00.855213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.855154 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-openshift-ca" pod "b0ef43af-1174-4cff-8176-72e91a86d6bf" (UID: "b0ef43af-1174-4cff-8176-72e91a86d6bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:00.855440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.855420 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-openshift-ca" pod "db7cbe18-b450-4519-8ce9-df81d1a3e943" (UID: "db7cbe18-b450-4519-8ce9-df81d1a3e943"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:00.855710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.855681 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.855800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.855709 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.857180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.857159 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7cbe18-b450-4519-8ce9-df81d1a3e943-kube-api-access-9jwnc" pod "db7cbe18-b450-4519-8ce9-df81d1a3e943" (UID: "db7cbe18-b450-4519-8ce9-df81d1a3e943"). InnerVolumeSpecName "kube-api-access-9jwnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:00.857409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.857389 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ef43af-1174-4cff-8176-72e91a86d6bf-kube-api-access-pvz7f" pod "b0ef43af-1174-4cff-8176-72e91a86d6bf" (UID: "b0ef43af-1174-4cff-8176-72e91a86d6bf"). InnerVolumeSpecName "kube-api-access-pvz7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:00.857515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.857484 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-xks-ca" pod "b0ef43af-1174-4cff-8176-72e91a86d6bf" (UID: "b0ef43af-1174-4cff-8176-72e91a86d6bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:00.857515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.857484 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-xks-ca" pod "db7cbe18-b450-4519-8ce9-df81d1a3e943" (UID: "db7cbe18-b450-4519-8ce9-df81d1a3e943"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:00.956924 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.956889 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-xks-ca\") pod \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " Nov 01 22:47:00.957169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.956951 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-kube-api-access-xkrjq\" (UniqueName: \"kubernetes.io/projected/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-kube-api-access-xkrjq\") pod \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " Nov 01 22:47:00.957169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.956988 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-openshift-ca\") pod \"6b1e2363-75e0-4edf-9171-d9166d9a340d\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " Nov 01 22:47:00.957169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.957077 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-xks-ca\") pod \"6b1e2363-75e0-4edf-9171-d9166d9a340d\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " Nov 01 22:47:00.957169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.957148 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-openshift-ca\") pod \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\" (UID: \"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1\") " Nov 01 22:47:00.957416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.957390 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1e2363-75e0-4edf-9171-d9166d9a340d-kube-api-access-2f6l6\" (UniqueName: \"kubernetes.io/projected/6b1e2363-75e0-4edf-9171-d9166d9a340d-kube-api-access-2f6l6\") pod \"6b1e2363-75e0-4edf-9171-d9166d9a340d\" (UID: \"6b1e2363-75e0-4edf-9171-d9166d9a340d\") " Nov 01 22:47:00.957563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.957527 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-openshift-ca" pod "a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1" (UID: "a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:00.957695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.957528 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-openshift-ca" pod "6b1e2363-75e0-4edf-9171-d9166d9a340d" (UID: "6b1e2363-75e0-4edf-9171-d9166d9a340d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:00.958250 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.958224 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.958350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.958255 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7cbe18-b450-4519-8ce9-df81d1a3e943-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.958350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.958273 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0ef43af-1174-4cff-8176-72e91a86d6bf-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.958350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.958288 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jwnc\" (UniqueName: \"kubernetes.io/projected/db7cbe18-b450-4519-8ce9-df81d1a3e943-kube-api-access-9jwnc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.958350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.958302 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.958350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.958318 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvz7f\" (UniqueName: \"kubernetes.io/projected/b0ef43af-1174-4cff-8176-72e91a86d6bf-kube-api-access-pvz7f\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.959499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.959471 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-xks-ca" pod "a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1" (UID: "a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:00.959499 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.959486 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1e2363-75e0-4edf-9171-d9166d9a340d-kube-api-access-2f6l6" pod "6b1e2363-75e0-4edf-9171-d9166d9a340d" (UID: "6b1e2363-75e0-4edf-9171-d9166d9a340d"). InnerVolumeSpecName "kube-api-access-2f6l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:00.959636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.959514 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-kube-api-access-xkrjq" pod "a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1" (UID: "a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1"). InnerVolumeSpecName "kube-api-access-xkrjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:00.959636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:00.959532 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-xks-ca" pod "6b1e2363-75e0-4edf-9171-d9166d9a340d" (UID: "6b1e2363-75e0-4edf-9171-d9166d9a340d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:01.059161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.059074 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1e2363-75e0-4edf-9171-d9166d9a340d-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:01.059161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.059101 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f6l6\" (UniqueName: \"kubernetes.io/projected/6b1e2363-75e0-4edf-9171-d9166d9a340d-kube-api-access-2f6l6\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:01.059161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.059140 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:01.059161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.059151 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkrjq\" (UniqueName: \"kubernetes.io/projected/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1-kube-api-access-xkrjq\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:01.344990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.344911 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4"] Nov 01 22:47:01.508539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.508470 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" path="/var/lib/kubelet/pods/bfad33b7-9986-496c-b4cd-b8b8d7a5bf83/volumes" Nov 01 22:47:01.534070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.534040 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" event={"ID":"6b1e2363-75e0-4edf-9171-d9166d9a340d","Type":"ContainerDied","Data":"becf5fd4e7e321c508854fbcf42fced4cca60afeace5207f7244703f73452b84"} Nov 01 22:47:01.534284 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.534271 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="becf5fd4e7e321c508854fbcf42fced4cca60afeace5207f7244703f73452b84" Nov 01 22:47:01.534424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.534415 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98" Nov 01 22:47:01.546028 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.545694 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" event={"ID":"b0ef43af-1174-4cff-8176-72e91a86d6bf","Type":"ContainerDied","Data":"465149aff52f222e074e732c980a3c2dc55040b158a119b9dae7ec3f7bfce263"} Nov 01 22:47:01.547494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.547061 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="465149aff52f222e074e732c980a3c2dc55040b158a119b9dae7ec3f7bfce263" Nov 01 22:47:01.547494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.546029 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx" Nov 01 22:47:01.549257 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.549232 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" event={"ID":"db7cbe18-b450-4519-8ce9-df81d1a3e943","Type":"ContainerDied","Data":"903a76f8d306c629de0bb08a2c7d63f4638fce89f481e7365d8cca796849285a"} Nov 01 22:47:01.549428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.549413 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="903a76f8d306c629de0bb08a2c7d63f4638fce89f481e7365d8cca796849285a" Nov 01 22:47:01.549987 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.549969 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz" Nov 01 22:47:01.562770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.562722 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" event={"ID":"a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1","Type":"ContainerDied","Data":"a18eb78f0452f6136cf24cf66e6df9f5e6d76fe8eb86462fadc5500cfb480d0b"} Nov 01 22:47:01.563072 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.563054 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a18eb78f0452f6136cf24cf66e6df9f5e6d76fe8eb86462fadc5500cfb480d0b" Nov 01 22:47:01.563297 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:01.563000 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz" Nov 01 22:47:03.635176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.597540 6287 generic.go:356] "Generic (PLEG): container finished" podID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerID="c8f0305aa2413a8f06b600611e0f04c0d3a83a7d8eb9e52527d4a6a57b07ba2d" exitCode=0 Nov 01 22:47:03.635176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.598591 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" event={"ID":"667699ea-ed8e-4055-ae52-3bd87e5e966d","Type":"ContainerDied","Data":"c8f0305aa2413a8f06b600611e0f04c0d3a83a7d8eb9e52527d4a6a57b07ba2d"} Nov 01 22:47:03.635176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.598653 6287 scope.go:122] "RemoveContainer" containerID="4d1d7b9a5725a3dafd8d44e347fb3a037c3e7a6c7797747054919431fc2982e7" Nov 01 22:47:03.635176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.606235 6287 generic.go:356] "Generic (PLEG): container finished" podID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerID="eaa2d2e65024d43527fcbd061f9d567d6dc903304ff41fc3378e7295c6427385" exitCode=0 Nov 01 22:47:03.635176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.606413 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" event={"ID":"a9a007c1-e42e-496b-b17a-fd657e79645d","Type":"ContainerDied","Data":"eaa2d2e65024d43527fcbd061f9d567d6dc903304ff41fc3378e7295c6427385"} Nov 01 22:47:03.635176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.610773 6287 generic.go:356] "Generic (PLEG): container finished" podID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerID="1c80a811a55c201d453f3f5b4f349cf9c31ff47adb110838d79102e0879a8c95" exitCode=0 Nov 01 22:47:03.635176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.610862 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" event={"ID":"9100bc64-fc11-4947-aaf9-5c762fd53ac7","Type":"ContainerDied","Data":"1c80a811a55c201d453f3f5b4f349cf9c31ff47adb110838d79102e0879a8c95"} Nov 01 22:47:03.781163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.781137 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.811704 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-token\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-token\") pod \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.811762 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-socket\") pod \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.811882 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-data\") pod \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.811918 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-podinfo\") pod \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.811965 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-kube-api-access-kk5b9\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-kube-api-access-kk5b9\") pod \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.812006 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istiod-ca-cert\") pod \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.812073 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-certs\") pod \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.812246 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-envoy\") pod \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.812295 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-credential-socket\") pod \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\" (UID: \"9100bc64-fc11-4947-aaf9-5c762fd53ac7\") " Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.813312 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-credential-socket" pod "9100bc64-fc11-4947-aaf9-5c762fd53ac7" (UID: "9100bc64-fc11-4947-aaf9-5c762fd53ac7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.813530 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-socket" pod "9100bc64-fc11-4947-aaf9-5c762fd53ac7" (UID: "9100bc64-fc11-4947-aaf9-5c762fd53ac7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.813713 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-data" pod "9100bc64-fc11-4947-aaf9-5c762fd53ac7" (UID: "9100bc64-fc11-4947-aaf9-5c762fd53ac7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.814306 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istiod-ca-cert" pod "9100bc64-fc11-4947-aaf9-5c762fd53ac7" (UID: "9100bc64-fc11-4947-aaf9-5c762fd53ac7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:03.816786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.815146 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-certs" pod "9100bc64-fc11-4947-aaf9-5c762fd53ac7" (UID: "9100bc64-fc11-4947-aaf9-5c762fd53ac7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.825997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.821991 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-envoy" pod "9100bc64-fc11-4947-aaf9-5c762fd53ac7" (UID: "9100bc64-fc11-4947-aaf9-5c762fd53ac7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.826590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.826546 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-token" pod "9100bc64-fc11-4947-aaf9-5c762fd53ac7" (UID: "9100bc64-fc11-4947-aaf9-5c762fd53ac7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.826740 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.826703 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-podinfo" pod "9100bc64-fc11-4947-aaf9-5c762fd53ac7" (UID: "9100bc64-fc11-4947-aaf9-5c762fd53ac7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:03.828222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.828191 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-kube-api-access-kk5b9" pod "9100bc64-fc11-4947-aaf9-5c762fd53ac7" (UID: "9100bc64-fc11-4947-aaf9-5c762fd53ac7"). InnerVolumeSpecName "kube-api-access-kk5b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.836924 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.836903 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:47:03.857452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.857340 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:47:03.913995 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.913958 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-socket\") pod \"a9a007c1-e42e-496b-b17a-fd657e79645d\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " Nov 01 22:47:03.914287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914004 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-envoy\") pod \"a9a007c1-e42e-496b-b17a-fd657e79645d\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " Nov 01 22:47:03.914287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914051 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-socket\") pod \"667699ea-ed8e-4055-ae52-3bd87e5e966d\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " Nov 01 22:47:03.914287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914073 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-kube-api-access-4cnnf\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-kube-api-access-4cnnf\") pod \"a9a007c1-e42e-496b-b17a-fd657e79645d\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " Nov 01 22:47:03.914287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914106 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-token\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-token\") pod \"667699ea-ed8e-4055-ae52-3bd87e5e966d\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " Nov 01 22:47:03.914287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914166 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-certs\") pod \"a9a007c1-e42e-496b-b17a-fd657e79645d\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " Nov 01 22:47:03.914593 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914456 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-socket" pod "667699ea-ed8e-4055-ae52-3bd87e5e966d" (UID: "667699ea-ed8e-4055-ae52-3bd87e5e966d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.914593 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914511 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-certs" pod "a9a007c1-e42e-496b-b17a-fd657e79645d" (UID: "a9a007c1-e42e-496b-b17a-fd657e79645d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.914683 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914609 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-kube-api-access-2pxdf\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-kube-api-access-2pxdf\") pod \"667699ea-ed8e-4055-ae52-3bd87e5e966d\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " Nov 01 22:47:03.914741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914692 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-socket" pod "a9a007c1-e42e-496b-b17a-fd657e79645d" (UID: "a9a007c1-e42e-496b-b17a-fd657e79645d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.914741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914694 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-credential-socket\") pod \"a9a007c1-e42e-496b-b17a-fd657e79645d\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " Nov 01 22:47:03.914863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914772 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-token\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-token\") pod \"a9a007c1-e42e-496b-b17a-fd657e79645d\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " Nov 01 22:47:03.914863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914845 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-data\") pod \"667699ea-ed8e-4055-ae52-3bd87e5e966d\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " Nov 01 22:47:03.914984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914914 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-credential-socket\") pod \"667699ea-ed8e-4055-ae52-3bd87e5e966d\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " Nov 01 22:47:03.914984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914933 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-credential-socket" pod "a9a007c1-e42e-496b-b17a-fd657e79645d" (UID: "a9a007c1-e42e-496b-b17a-fd657e79645d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.914984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914955 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a007c1-e42e-496b-b17a-fd657e79645d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a007c1-e42e-496b-b17a-fd657e79645d-istiod-ca-cert\") pod \"a9a007c1-e42e-496b-b17a-fd657e79645d\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " Nov 01 22:47:03.915186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.914988 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-certs\") pod \"667699ea-ed8e-4055-ae52-3bd87e5e966d\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " Nov 01 22:47:03.915186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915032 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-envoy\") pod \"667699ea-ed8e-4055-ae52-3bd87e5e966d\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " Nov 01 22:47:03.915186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915061 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-data\") pod \"a9a007c1-e42e-496b-b17a-fd657e79645d\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " Nov 01 22:47:03.915186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915146 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-podinfo\") pod \"a9a007c1-e42e-496b-b17a-fd657e79645d\" (UID: \"a9a007c1-e42e-496b-b17a-fd657e79645d\") " Nov 01 22:47:03.915415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915187 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-data" pod "667699ea-ed8e-4055-ae52-3bd87e5e966d" (UID: "667699ea-ed8e-4055-ae52-3bd87e5e966d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.915415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915189 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-podinfo\") pod \"667699ea-ed8e-4055-ae52-3bd87e5e966d\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " Nov 01 22:47:03.915415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915247 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/667699ea-ed8e-4055-ae52-3bd87e5e966d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667699ea-ed8e-4055-ae52-3bd87e5e966d-istiod-ca-cert\") pod \"667699ea-ed8e-4055-ae52-3bd87e5e966d\" (UID: \"667699ea-ed8e-4055-ae52-3bd87e5e966d\") " Nov 01 22:47:03.915721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915683 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-certs" pod "667699ea-ed8e-4055-ae52-3bd87e5e966d" (UID: "667699ea-ed8e-4055-ae52-3bd87e5e966d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.915899 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915878 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-credential-socket" pod "667699ea-ed8e-4055-ae52-3bd87e5e966d" (UID: "667699ea-ed8e-4055-ae52-3bd87e5e966d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.915972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915932 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.915972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915954 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915969 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.915986 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916001 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916014 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916027 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916040 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916054 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916069 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk5b9\" (UniqueName: \"kubernetes.io/projected/9100bc64-fc11-4947-aaf9-5c762fd53ac7-kube-api-access-kk5b9\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916084 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916098 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916132 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916148 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9100bc64-fc11-4947-aaf9-5c762fd53ac7-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916161 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:03.916621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916199 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a007c1-e42e-496b-b17a-fd657e79645d-istiod-ca-cert" pod "a9a007c1-e42e-496b-b17a-fd657e79645d" (UID: "a9a007c1-e42e-496b-b17a-fd657e79645d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:03.916621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916475 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667699ea-ed8e-4055-ae52-3bd87e5e966d-istiod-ca-cert" pod "667699ea-ed8e-4055-ae52-3bd87e5e966d" (UID: "667699ea-ed8e-4055-ae52-3bd87e5e966d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:03.916621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.916495 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-data" pod "a9a007c1-e42e-496b-b17a-fd657e79645d" (UID: "a9a007c1-e42e-496b-b17a-fd657e79645d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.917863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.917829 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-envoy" pod "a9a007c1-e42e-496b-b17a-fd657e79645d" (UID: "a9a007c1-e42e-496b-b17a-fd657e79645d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:03.918180 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.918157 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-token" pod "a9a007c1-e42e-496b-b17a-fd657e79645d" (UID: "a9a007c1-e42e-496b-b17a-fd657e79645d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.918283 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.918194 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-kube-api-access-2pxdf" pod "667699ea-ed8e-4055-ae52-3bd87e5e966d" (UID: "667699ea-ed8e-4055-ae52-3bd87e5e966d"). InnerVolumeSpecName "kube-api-access-2pxdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.918627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.918602 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-token" pod "667699ea-ed8e-4055-ae52-3bd87e5e966d" (UID: "667699ea-ed8e-4055-ae52-3bd87e5e966d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.918715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.918639 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-kube-api-access-4cnnf" pod "a9a007c1-e42e-496b-b17a-fd657e79645d" (UID: "a9a007c1-e42e-496b-b17a-fd657e79645d"). InnerVolumeSpecName "kube-api-access-4cnnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:03.918876 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.918860 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-podinfo" pod "667699ea-ed8e-4055-ae52-3bd87e5e966d" (UID: "667699ea-ed8e-4055-ae52-3bd87e5e966d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:03.919044 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.919029 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-podinfo" pod "a9a007c1-e42e-496b-b17a-fd657e79645d" (UID: "a9a007c1-e42e-496b-b17a-fd657e79645d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:03.919839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:03.919816 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-envoy" pod "667699ea-ed8e-4055-ae52-3bd87e5e966d" (UID: "667699ea-ed8e-4055-ae52-3bd87e5e966d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.017515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017484 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017514 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9a007c1-e42e-496b-b17a-fd657e79645d-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017525 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017536 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017546 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017554 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017563 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667699ea-ed8e-4055-ae52-3bd87e5e966d-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017571 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017583 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnnf\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-kube-api-access-4cnnf\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017592 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017601 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pxdf\" (UniqueName: \"kubernetes.io/projected/667699ea-ed8e-4055-ae52-3bd87e5e966d-kube-api-access-2pxdf\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.017719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.017609 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9a007c1-e42e-496b-b17a-fd657e79645d-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.067404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.067367 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" podUID="b00ff221-8630-4a3a-b9c3-37e90a1394fa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:04.067568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.067367 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" podUID="f58e1b66-6f12-4a7b-93d4-02c43e499f7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:04.081506 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.081456 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" containerID="cri-o://1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e" gracePeriod=45 Nov 01 22:47:04.093082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.093056 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" containerID="cri-o://47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16" gracePeriod=45 Nov 01 22:47:04.384142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.380440 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.427540 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-model-cache\") pod \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.427614 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-kube-api-access-qlscn\" (UniqueName: \"kubernetes.io/projected/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-kube-api-access-qlscn\") pod \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.427693 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tmp-dir\") pod \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.427740 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-home\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-home\") pod \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.428011 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-home" pod "1263f31e-87e9-4a4a-9f92-f0a03d3e0095" (UID: "1263f31e-87e9-4a4a-9f92-f0a03d3e0095"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.428164 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-model-cache" pod "1263f31e-87e9-4a4a-9f92-f0a03d3e0095" (UID: "1263f31e-87e9-4a4a-9f92-f0a03d3e0095"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.428250 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tmp-dir" pod "1263f31e-87e9-4a4a-9f92-f0a03d3e0095" (UID: "1263f31e-87e9-4a4a-9f92-f0a03d3e0095"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.428354 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-dshm\") pod \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.428391 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tls-certs\" (UniqueName: \"kubernetes.io/secret/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tls-certs\") pod \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\" (UID: \"1263f31e-87e9-4a4a-9f92-f0a03d3e0095\") " Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.428803 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.428816 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.429267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.428825 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.435150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.431488 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-kube-api-access-qlscn" pod "1263f31e-87e9-4a4a-9f92-f0a03d3e0095" (UID: "1263f31e-87e9-4a4a-9f92-f0a03d3e0095"). InnerVolumeSpecName "kube-api-access-qlscn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:04.435150 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.431495 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tls-certs" pod "1263f31e-87e9-4a4a-9f92-f0a03d3e0095" (UID: "1263f31e-87e9-4a4a-9f92-f0a03d3e0095"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:04.435346 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.435102 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-dshm" pod "1263f31e-87e9-4a4a-9f92-f0a03d3e0095" (UID: "1263f31e-87e9-4a4a-9f92-f0a03d3e0095"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.436522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.436185 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:47:04.530033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.530001 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tmp-dir\") pod \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " Nov 01 22:47:04.530200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.530086 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-model-cache\") pod \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " Nov 01 22:47:04.530200 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.530154 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-home\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-home\") pod \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " Nov 01 22:47:04.530338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.530284 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-dshm\") pod \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " Nov 01 22:47:04.530406 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.530326 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tmp-dir" pod "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" (UID: "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.530406 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.530375 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-kube-api-access-g4lzh\" (UniqueName: \"kubernetes.io/projected/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-kube-api-access-g4lzh\") pod \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " Nov 01 22:47:04.530507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.530407 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-model-cache" pod "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" (UID: "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.530507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.530427 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tls-certs\") pod \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\" (UID: \"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8\") " Nov 01 22:47:04.530507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.530429 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-home" pod "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" (UID: "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.531214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.531061 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.531214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.531083 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.531214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.531096 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.531214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.531151 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlscn\" (UniqueName: \"kubernetes.io/projected/1263f31e-87e9-4a4a-9f92-f0a03d3e0095-kube-api-access-qlscn\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.531214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.531163 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.531214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.531176 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.533143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.533105 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tls-certs" pod "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" (UID: "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:04.533266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.533250 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-dshm" pod "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" (UID: "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:04.533532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.533518 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-kube-api-access-g4lzh" pod "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" (UID: "f2cab38c-1368-4340-acd0-5a6d5dbd4ab8"). InnerVolumeSpecName "kube-api-access-g4lzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:04.621494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.621467 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" Nov 01 22:47:04.621660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.621511 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6" event={"ID":"667699ea-ed8e-4055-ae52-3bd87e5e966d","Type":"ContainerDied","Data":"ef7b718aa95dee62590ed0ac09a1b5da935680051b502f513a6095e79dc3f25b"} Nov 01 22:47:04.621660 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.621572 6287 scope.go:122] "RemoveContainer" containerID="c8f0305aa2413a8f06b600611e0f04c0d3a83a7d8eb9e52527d4a6a57b07ba2d" Nov 01 22:47:04.623012 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.622995 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" Nov 01 22:47:04.623080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.623046 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f" event={"ID":"a9a007c1-e42e-496b-b17a-fd657e79645d","Type":"ContainerDied","Data":"22d421edcdc50417c27d8a6b5486a8f1e5c6aa4ddab1db1fac4503a6667e37e3"} Nov 01 22:47:04.625008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.624962 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" Nov 01 22:47:04.625193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.625000 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5" event={"ID":"9100bc64-fc11-4947-aaf9-5c762fd53ac7","Type":"ContainerDied","Data":"87e76022f2f2caca53f68a53b310c794613a603841bc11f6f092969fa7016642"} Nov 01 22:47:04.627567 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.627545 6287 generic.go:356] "Generic (PLEG): container finished" podID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerID="1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e" exitCode=0 Nov 01 22:47:04.627698 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.627591 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" Nov 01 22:47:04.627698 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.627637 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" event={"ID":"1263f31e-87e9-4a4a-9f92-f0a03d3e0095","Type":"ContainerDied","Data":"1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e"} Nov 01 22:47:04.627698 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.627668 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn" event={"ID":"1263f31e-87e9-4a4a-9f92-f0a03d3e0095","Type":"ContainerDied","Data":"eaff3f3ed3e81e24a5f655292ca716742c0848417ca139f6492a9fd2bacd17ee"} Nov 01 22:47:04.629403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.629376 6287 generic.go:356] "Generic (PLEG): container finished" podID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerID="47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16" exitCode=0 Nov 01 22:47:04.629525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.629420 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" event={"ID":"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8","Type":"ContainerDied","Data":"47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16"} Nov 01 22:47:04.629525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.629445 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" event={"ID":"f2cab38c-1368-4340-acd0-5a6d5dbd4ab8","Type":"ContainerDied","Data":"0ec0a574d1cd29040e9efe8f1b78f8a4a304a1986a41be19c9d869715faef631"} Nov 01 22:47:04.629525 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.629453 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6" Nov 01 22:47:04.632566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.632545 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.632653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.632570 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4lzh\" (UniqueName: \"kubernetes.io/projected/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-kube-api-access-g4lzh\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.632653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.632585 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:04.636279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.636258 6287 scope.go:122] "RemoveContainer" containerID="eaa2d2e65024d43527fcbd061f9d567d6dc903304ff41fc3378e7295c6427385" Nov 01 22:47:04.647902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.647885 6287 scope.go:122] "RemoveContainer" containerID="1c80a811a55c201d453f3f5b4f349cf9c31ff47adb110838d79102e0879a8c95" Nov 01 22:47:04.657980 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.657960 6287 scope.go:122] "RemoveContainer" containerID="1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e" Nov 01 22:47:04.667032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.667006 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f"] Nov 01 22:47:04.672923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.672898 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-zvq5f"] Nov 01 22:47:04.674210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.674194 6287 scope.go:122] "RemoveContainer" containerID="1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e" Nov 01 22:47:04.674534 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:04.674515 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e\": container with ID starting with 1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e not found: ID does not exist" containerID="1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e" Nov 01 22:47:04.674601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.674546 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e"} err="failed to get container status \"1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e\": rpc error: code = NotFound desc = could not find container \"1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e\": container with ID starting with 1017c5cc26317f1f54b58cbe9ec98520bdb6a71650012b3bdd80a53b119c899e not found: ID does not exist" Nov 01 22:47:04.674601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.674570 6287 scope.go:122] "RemoveContainer" containerID="47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16" Nov 01 22:47:04.680345 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.680323 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6"] Nov 01 22:47:04.686036 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.686019 6287 scope.go:122] "RemoveContainer" containerID="47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16" Nov 01 22:47:04.686279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.686262 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-htcm6"] Nov 01 22:47:04.686353 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:04.686329 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16\": container with ID starting with 47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16 not found: ID does not exist" containerID="47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16" Nov 01 22:47:04.686419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.686363 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16"} err="failed to get container status \"47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16\": rpc error: code = NotFound desc = could not find container \"47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16\": container with ID starting with 47978f8334ce47df85c6f5fd1bf87d192b0953a54a68bcd971577c2fb5f47b16 not found: ID does not exist" Nov 01 22:47:04.699743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.699712 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn"] Nov 01 22:47:04.704904 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.704881 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-381659/premium-w1-381659-kserve-657d99b85f-9xjpn"] Nov 01 22:47:04.724967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.724933 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6"] Nov 01 22:47:04.731869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.731842 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-381659/simulator-w1-381659-kserve-65cb6cb7c8-j9jd6"] Nov 01 22:47:04.753614 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.753587 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5"] Nov 01 22:47:04.765666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:04.765639 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-00e49d-openshift-default-5bd6688d47-2fdv5"] Nov 01 22:47:05.087788 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:05.087753 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" podUID="5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:05.485409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:05.485325 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" path="/var/lib/kubelet/pods/1263f31e-87e9-4a4a-9f92-f0a03d3e0095/volumes" Nov 01 22:47:05.487037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:05.487008 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" path="/var/lib/kubelet/pods/667699ea-ed8e-4055-ae52-3bd87e5e966d/volumes" Nov 01 22:47:05.487647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:05.487616 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" path="/var/lib/kubelet/pods/9100bc64-fc11-4947-aaf9-5c762fd53ac7/volumes" Nov 01 22:47:05.488572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:05.488542 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" path="/var/lib/kubelet/pods/a9a007c1-e42e-496b-b17a-fd657e79645d/volumes" Nov 01 22:47:05.489564 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:05.489543 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" path="/var/lib/kubelet/pods/f2cab38c-1368-4340-acd0-5a6d5dbd4ab8/volumes" Nov 01 22:47:05.599281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:05.599244 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775"] Nov 01 22:47:05.602522 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:05.602498 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3fa400-ht775"] Nov 01 22:47:06.345446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.345400 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" podUID="05897fec-6ded-4175-9ead-46d45ce818f8" containerName="payload-pre-processing" containerID="cri-o://d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025" gracePeriod=25 Nov 01 22:47:06.353266 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:47:06.353242 6287 logging.go:55] [core] [Channel #4336 SubChannel #4337]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.169:50052", ServerName: "10.133.0.169:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.169:50052: connect: connection refused" Nov 01 22:47:06.631462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.631436 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:47:06.662803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.662769 6287 generic.go:356] "Generic (PLEG): container finished" podID="05897fec-6ded-4175-9ead-46d45ce818f8" containerID="d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025" exitCode=0 Nov 01 22:47:06.662976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.662820 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" Nov 01 22:47:06.662976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.662850 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" event={"ID":"05897fec-6ded-4175-9ead-46d45ce818f8","Type":"ContainerDied","Data":"d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025"} Nov 01 22:47:06.662976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.662926 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" event={"ID":"05897fec-6ded-4175-9ead-46d45ce818f8","Type":"ContainerDied","Data":"9e3a6e29dfeb2eb747a659799cef4439ad14f9983955acea8145c0a6a690d795"} Nov 01 22:47:06.662976 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.662958 6287 scope.go:122] "RemoveContainer" containerID="d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025" Nov 01 22:47:06.666480 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.666451 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05897fec-6ded-4175-9ead-46d45ce818f8-kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/05897fec-6ded-4175-9ead-46d45ce818f8-kube-api-access-4vlvg\") pod \"05897fec-6ded-4175-9ead-46d45ce818f8\" (UID: \"05897fec-6ded-4175-9ead-46d45ce818f8\") " Nov 01 22:47:06.666595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.666545 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05897fec-6ded-4175-9ead-46d45ce818f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05897fec-6ded-4175-9ead-46d45ce818f8-plugins-config-volume\") pod \"05897fec-6ded-4175-9ead-46d45ce818f8\" (UID: \"05897fec-6ded-4175-9ead-46d45ce818f8\") " Nov 01 22:47:06.667551 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.667497 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05897fec-6ded-4175-9ead-46d45ce818f8-plugins-config-volume" pod "05897fec-6ded-4175-9ead-46d45ce818f8" (UID: "05897fec-6ded-4175-9ead-46d45ce818f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:06.669741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.669715 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05897fec-6ded-4175-9ead-46d45ce818f8-kube-api-access-4vlvg" pod "05897fec-6ded-4175-9ead-46d45ce818f8" (UID: "05897fec-6ded-4175-9ead-46d45ce818f8"). InnerVolumeSpecName "kube-api-access-4vlvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:06.680807 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.680788 6287 scope.go:122] "RemoveContainer" containerID="d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025" Nov 01 22:47:06.681053 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:06.681032 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025\": container with ID starting with d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025 not found: ID does not exist" containerID="d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025" Nov 01 22:47:06.681142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.681061 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025"} err="failed to get container status \"d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025\": rpc error: code = NotFound desc = could not find container \"d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025\": container with ID starting with d27a277fc24f7e75d9588862533cecc73d16af68d1f5c98bae3cd9a795beb025 not found: ID does not exist" Nov 01 22:47:06.768872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.768841 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/05897fec-6ded-4175-9ead-46d45ce818f8-kube-api-access-4vlvg\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:06.768872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.768869 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05897fec-6ded-4175-9ead-46d45ce818f8-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:06.997141 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:06.997083 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4"] Nov 01 22:47:07.000535 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:07.000506 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4"] Nov 01 22:47:07.353884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:07.353842 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3fa400-5d45c86b6c-jrfv4" podUID="05897fec-6ded-4175-9ead-46d45ce818f8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.169:50052\" within 1s: context deadline exceeded" Nov 01 22:47:07.473211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:07.473181 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05897fec-6ded-4175-9ead-46d45ce818f8" path="/var/lib/kubelet/pods/05897fec-6ded-4175-9ead-46d45ce818f8/volumes" Nov 01 22:47:07.473612 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:07.473599 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670e10db-0c78-44ab-bb15-c445aae81921" path="/var/lib/kubelet/pods/670e10db-0c78-44ab-bb15-c445aae81921/volumes" Nov 01 22:47:09.615922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:09.615886 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8"] Nov 01 22:47:09.621587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:09.621560 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-381659-sf6b8"] Nov 01 22:47:10.193549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:10.193506 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7"] Nov 01 22:47:10.199703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:10.199677 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3d8490-cxsn7"] Nov 01 22:47:11.470695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.470664 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d098ef1-2b4a-4299-a4b9-24d665b34f93" path="/var/lib/kubelet/pods/6d098ef1-2b4a-4299-a4b9-24d665b34f93/volumes" Nov 01 22:47:11.471088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.471073 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cfd83d7-80ae-491b-998f-f25199c72bf4" path="/var/lib/kubelet/pods/7cfd83d7-80ae-491b-998f-f25199c72bf4/volumes" Nov 01 22:47:11.741056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.741025 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:47:11.744312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.744279 6287 generic.go:356] "Generic (PLEG): container finished" podID="b00ff221-8630-4a3a-b9c3-37e90a1394fa" containerID="33a300b560ac797a0c7c18a2c6dbe9e46ed51f480f13f301adfb08e88d537c97" exitCode=0 Nov 01 22:47:11.744448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.744353 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" event={"ID":"b00ff221-8630-4a3a-b9c3-37e90a1394fa","Type":"ContainerDied","Data":"33a300b560ac797a0c7c18a2c6dbe9e46ed51f480f13f301adfb08e88d537c97"} Nov 01 22:47:11.748625 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.748580 6287 generic.go:356] "Generic (PLEG): container finished" podID="5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" containerID="124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748" exitCode=0 Nov 01 22:47:11.748625 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.748615 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" event={"ID":"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a","Type":"ContainerDied","Data":"124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748"} Nov 01 22:47:11.748625 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.748628 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" Nov 01 22:47:11.748984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.748657 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn" event={"ID":"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a","Type":"ContainerDied","Data":"41a434a9efd06d5c485a4440b427a45e6c6ce934d969380f89f18ad317dacec5"} Nov 01 22:47:11.748984 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.748686 6287 scope.go:122] "RemoveContainer" containerID="124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748" Nov 01 22:47:11.763572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.763550 6287 scope.go:122] "RemoveContainer" containerID="124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748" Nov 01 22:47:11.763909 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:11.763879 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748\": container with ID starting with 124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748 not found: ID does not exist" containerID="124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748" Nov 01 22:47:11.764016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.763923 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748"} err="failed to get container status \"124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748\": rpc error: code = NotFound desc = could not find container \"124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748\": container with ID starting with 124b8d30565e14b073687f22b01b2014d413e33b2fdad555a21dbb01219d6748 not found: ID does not exist" Nov 01 22:47:11.803334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.803302 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:47:11.840000 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.839968 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00ff221-8630-4a3a-b9c3-37e90a1394fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00ff221-8630-4a3a-b9c3-37e90a1394fa-plugins-config-volume\") pod \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\" (UID: \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\") " Nov 01 22:47:11.840172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.840010 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-kube-api-access-27fls\" (UniqueName: \"kubernetes.io/projected/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-kube-api-access-27fls\") pod \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\" (UID: \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\") " Nov 01 22:47:11.840172 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.840072 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-plugins-config-volume\") pod \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\" (UID: \"5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a\") " Nov 01 22:47:11.840294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.840258 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00ff221-8630-4a3a-b9c3-37e90a1394fa-kube-api-access-dddlg\" (UniqueName: \"kubernetes.io/projected/b00ff221-8630-4a3a-b9c3-37e90a1394fa-kube-api-access-dddlg\") pod \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\" (UID: \"b00ff221-8630-4a3a-b9c3-37e90a1394fa\") " Nov 01 22:47:11.840587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.840565 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00ff221-8630-4a3a-b9c3-37e90a1394fa-plugins-config-volume" pod "b00ff221-8630-4a3a-b9c3-37e90a1394fa" (UID: "b00ff221-8630-4a3a-b9c3-37e90a1394fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:11.840656 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.840593 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-plugins-config-volume" pod "5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" (UID: "5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:11.840818 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.840796 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00ff221-8630-4a3a-b9c3-37e90a1394fa-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:11.840903 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.840820 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:11.842537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.842515 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00ff221-8630-4a3a-b9c3-37e90a1394fa-kube-api-access-dddlg" pod "b00ff221-8630-4a3a-b9c3-37e90a1394fa" (UID: "b00ff221-8630-4a3a-b9c3-37e90a1394fa"). InnerVolumeSpecName "kube-api-access-dddlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:11.842630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.842549 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-kube-api-access-27fls" pod "5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" (UID: "5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a"). InnerVolumeSpecName "kube-api-access-27fls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:11.942430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.942392 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dddlg\" (UniqueName: \"kubernetes.io/projected/b00ff221-8630-4a3a-b9c3-37e90a1394fa-kube-api-access-dddlg\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:11.942430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:11.942425 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27fls\" (UniqueName: \"kubernetes.io/projected/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a-kube-api-access-27fls\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:12.075542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:12.075511 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn"] Nov 01 22:47:12.083010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:12.082981 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3d8490-767b564c79-5wjgn"] Nov 01 22:47:12.759543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:12.759512 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" Nov 01 22:47:12.760030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:12.759538 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt" event={"ID":"b00ff221-8630-4a3a-b9c3-37e90a1394fa","Type":"ContainerDied","Data":"5883d6b14f2da8c6b8a60bb5b153eea3916513c9b57e9d2a46835a6dbc954458"} Nov 01 22:47:12.760030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:12.759620 6287 scope.go:122] "RemoveContainer" containerID="33a300b560ac797a0c7c18a2c6dbe9e46ed51f480f13f301adfb08e88d537c97" Nov 01 22:47:12.790554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:12.790494 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt"] Nov 01 22:47:12.793915 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:12.793887 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3d8490-6d4f68475f-82mpt"] Nov 01 22:47:13.471565 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:13.471531 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" path="/var/lib/kubelet/pods/5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a/volumes" Nov 01 22:47:13.471928 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:13.471915 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00ff221-8630-4a3a-b9c3-37e90a1394fa" path="/var/lib/kubelet/pods/b00ff221-8630-4a3a-b9c3-37e90a1394fa/volumes" Nov 01 22:47:14.067335 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.067301 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" podUID="f58e1b66-6f12-4a7b-93d4-02c43e499f7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:47:14.896091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896057 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458"] Nov 01 22:47:14.896561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896548 6287 cpu_manager.go:414] "containerMap: removing container" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" Nov 01 22:47:14.896627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896562 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" Nov 01 22:47:14.896627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896576 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1" containerName="revoke-keys" Nov 01 22:47:14.896627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896583 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1" containerName="revoke-keys" Nov 01 22:47:14.896627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896601 6287 cpu_manager.go:414] "containerMap: removing container" podUID="6b1e2363-75e0-4edf-9171-d9166d9a340d" containerName="revoke-keys" Nov 01 22:47:14.896627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896608 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1e2363-75e0-4edf-9171-d9166d9a340d" containerName="revoke-keys" Nov 01 22:47:14.896627 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896621 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" containerName="payload-pre-processing" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896629 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" containerName="payload-pre-processing" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896637 6287 cpu_manager.go:414] "containerMap: removing container" podUID="05897fec-6ded-4175-9ead-46d45ce818f8" containerName="payload-pre-processing" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896642 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05897fec-6ded-4175-9ead-46d45ce818f8" containerName="payload-pre-processing" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896660 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896664 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896674 6287 cpu_manager.go:414] "containerMap: removing container" podUID="db7cbe18-b450-4519-8ce9-df81d1a3e943" containerName="revoke-keys" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896678 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db7cbe18-b450-4519-8ce9-df81d1a3e943" containerName="revoke-keys" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896686 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896691 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896700 6287 cpu_manager.go:414] "containerMap: removing container" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896704 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896711 6287 cpu_manager.go:414] "containerMap: removing container" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896716 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896723 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b0ef43af-1174-4cff-8176-72e91a86d6bf" containerName="revoke-keys" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896728 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ef43af-1174-4cff-8176-72e91a86d6bf" containerName="revoke-keys" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896739 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b00ff221-8630-4a3a-b9c3-37e90a1394fa" containerName="payload-processing" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896744 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00ff221-8630-4a3a-b9c3-37e90a1394fa" containerName="payload-processing" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896750 6287 cpu_manager.go:414] "containerMap: removing container" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" Nov 01 22:47:14.896827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896755 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896843 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="1263f31e-87e9-4a4a-9f92-f0a03d3e0095" containerName="main" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896853 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfad33b7-9986-496c-b4cd-b8b8d7a5bf83" containerName="istio-proxy" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896859 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ef43af-1174-4cff-8176-72e91a86d6bf" containerName="revoke-keys" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896866 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896871 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00ff221-8630-4a3a-b9c3-37e90a1394fa" containerName="payload-processing" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896878 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9100bc64-fc11-4947-aaf9-5c762fd53ac7" containerName="istio-proxy" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896885 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1" containerName="revoke-keys" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896892 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1e2363-75e0-4edf-9171-d9166d9a340d" containerName="revoke-keys" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896898 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3f18bd-9b22-49cc-b6b9-5ad5ed6a014a" containerName="payload-pre-processing" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896903 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="05897fec-6ded-4175-9ead-46d45ce818f8" containerName="payload-pre-processing" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896911 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2cab38c-1368-4340-acd0-5a6d5dbd4ab8" containerName="main" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896917 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="db7cbe18-b450-4519-8ce9-df81d1a3e943" containerName="revoke-keys" Nov 01 22:47:14.897409 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.896923 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a007c1-e42e-496b-b17a-fd657e79645d" containerName="istio-proxy" Nov 01 22:47:14.901905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.901883 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:14.907952 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.907928 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:47:14.908658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.908639 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:47:14.918926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.918898 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458"] Nov 01 22:47:14.958680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.958655 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:47:14.958680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.958655 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:47:14.993960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.993928 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:14.994096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.993970 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:14.994096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:14.994022 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m2bb\" (UniqueName: \"kubernetes.io/projected/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-kube-api-access-5m2bb\") pod \"maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:15.095538 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:15.095503 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m2bb\" (UniqueName: \"kubernetes.io/projected/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-kube-api-access-5m2bb\") pod \"maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:15.095884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:15.095718 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:15.095884 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:15.095765 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:15.096455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:15.096437 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:15.098157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:15.098101 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:15.107581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:15.107546 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m2bb\" (UniqueName: \"kubernetes.io/projected/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-kube-api-access-5m2bb\") pod \"maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:15.216146 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:15.216029 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:15.522268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:15.522238 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458"] Nov 01 22:47:15.523651 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:47:15.523625 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e8b716_bb96_4a9b_bc0d_43f1fddb02ed.slice/crio-8ba02d4f1500c7b29beb065dab12ccbcd9bb709717db106dc057033aa47393c0 WatchSource:0}: Error finding container 8ba02d4f1500c7b29beb065dab12ccbcd9bb709717db106dc057033aa47393c0: Status 404 returned error can't find the container with id 8ba02d4f1500c7b29beb065dab12ccbcd9bb709717db106dc057033aa47393c0 Nov 01 22:47:15.589751 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:15.589711 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:47:15.788165 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:15.788135 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" event={"ID":"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed","Type":"ContainerStarted","Data":"8ba02d4f1500c7b29beb065dab12ccbcd9bb709717db106dc057033aa47393c0"} Nov 01 22:47:16.799179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:16.799088 6287 generic.go:356] "Generic (PLEG): container finished" podID="75e8b716-bb96-4a9b-bc0d-43f1fddb02ed" containerID="ad69bb190822825c349f11f3f28840b3d3c4b0cb06f336458bb8d00f03c25716" exitCode=0 Nov 01 22:47:16.799179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:16.799166 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" event={"ID":"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed","Type":"ContainerDied","Data":"ad69bb190822825c349f11f3f28840b3d3c4b0cb06f336458bb8d00f03c25716"} Nov 01 22:47:18.381308 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.381274 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:18.451637 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.451603 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-kube-api-access-5m2bb\" (UniqueName: \"kubernetes.io/projected/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-kube-api-access-5m2bb\") pod \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " Nov 01 22:47:18.451885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.451867 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-xks-ca\") pod \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " Nov 01 22:47:18.452250 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.452229 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-openshift-ca\") pod \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\" (UID: \"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed\") " Nov 01 22:47:18.453170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.453146 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-openshift-ca" pod "75e8b716-bb96-4a9b-bc0d-43f1fddb02ed" (UID: "75e8b716-bb96-4a9b-bc0d-43f1fddb02ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:18.457543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.457519 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-kube-api-access-5m2bb" pod "75e8b716-bb96-4a9b-bc0d-43f1fddb02ed" (UID: "75e8b716-bb96-4a9b-bc0d-43f1fddb02ed"). InnerVolumeSpecName "kube-api-access-5m2bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:18.459312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.459288 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-xks-ca" pod "75e8b716-bb96-4a9b-bc0d-43f1fddb02ed" (UID: "75e8b716-bb96-4a9b-bc0d-43f1fddb02ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:18.554853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.554761 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:18.555087 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.555067 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m2bb\" (UniqueName: \"kubernetes.io/projected/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-kube-api-access-5m2bb\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:18.555205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.555191 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:18.877190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.876254 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" event={"ID":"75e8b716-bb96-4a9b-bc0d-43f1fddb02ed","Type":"ContainerDied","Data":"8ba02d4f1500c7b29beb065dab12ccbcd9bb709717db106dc057033aa47393c0"} Nov 01 22:47:18.877190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.876323 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ba02d4f1500c7b29beb065dab12ccbcd9bb709717db106dc057033aa47393c0" Nov 01 22:47:18.877190 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:18.876397 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458" Nov 01 22:47:19.736017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.735991 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:47:19.891059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.891026 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-kube-api-access-dp5wc\" (UniqueName: \"kubernetes.io/projected/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-kube-api-access-dp5wc\") pod \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\" (UID: \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\") " Nov 01 22:47:19.891255 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.891237 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-plugins-config-volume\") pod \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\" (UID: \"f58e1b66-6f12-4a7b-93d4-02c43e499f7a\") " Nov 01 22:47:19.891620 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.891594 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-plugins-config-volume" pod "f58e1b66-6f12-4a7b-93d4-02c43e499f7a" (UID: "f58e1b66-6f12-4a7b-93d4-02c43e499f7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:19.891804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.891781 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:19.893432 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.893401 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-kube-api-access-dp5wc" pod "f58e1b66-6f12-4a7b-93d4-02c43e499f7a" (UID: "f58e1b66-6f12-4a7b-93d4-02c43e499f7a"). InnerVolumeSpecName "kube-api-access-dp5wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:19.894398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.894373 6287 generic.go:356] "Generic (PLEG): container finished" podID="f58e1b66-6f12-4a7b-93d4-02c43e499f7a" containerID="8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd" exitCode=0 Nov 01 22:47:19.894483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.894417 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" Nov 01 22:47:19.894483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.894450 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" event={"ID":"f58e1b66-6f12-4a7b-93d4-02c43e499f7a","Type":"ContainerDied","Data":"8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd"} Nov 01 22:47:19.894571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.894507 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs" event={"ID":"f58e1b66-6f12-4a7b-93d4-02c43e499f7a","Type":"ContainerDied","Data":"f658bb8bd89064f2882bdfc21945568ee488a628acec66bf923e76e532094df3"} Nov 01 22:47:19.894571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.894536 6287 scope.go:122] "RemoveContainer" containerID="8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd" Nov 01 22:47:19.912357 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.912335 6287 scope.go:122] "RemoveContainer" containerID="8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd" Nov 01 22:47:19.912689 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:19.912669 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd\": container with ID starting with 8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd not found: ID does not exist" containerID="8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd" Nov 01 22:47:19.912755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.912699 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd"} err="failed to get container status \"8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd\": rpc error: code = NotFound desc = could not find container \"8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd\": container with ID starting with 8574931dc30dfc2fb981aec2548d843300ed4970d832847a194af16c4056f4dd not found: ID does not exist" Nov 01 22:47:19.925546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.925522 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs"] Nov 01 22:47:19.934338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.934311 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl"] Nov 01 22:47:19.941010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.940981 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-381659-7f59b45b6f-d7mhs"] Nov 01 22:47:19.993177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:19.993108 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp5wc\" (UniqueName: \"kubernetes.io/projected/f58e1b66-6f12-4a7b-93d4-02c43e499f7a-kube-api-access-dp5wc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:20.315151 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:20.315043 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls"] Nov 01 22:47:20.315320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:20.315297 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" containerID="cri-o://2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184" gracePeriod=30 Nov 01 22:47:20.347476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:20.347440 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226"] Nov 01 22:47:20.347680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:20.347656 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" containerID="cri-o://a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490" gracePeriod=30 Nov 01 22:47:21.234972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:21.234936 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr"] Nov 01 22:47:21.235547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:21.235220 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" containerID="cri-o://546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075" gracePeriod=30 Nov 01 22:47:21.239602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:21.239564 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg"] Nov 01 22:47:21.240092 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:21.239985 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" containerID="cri-o://f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd" gracePeriod=30 Nov 01 22:47:21.471959 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:21.471927 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58e1b66-6f12-4a7b-93d4-02c43e499f7a" path="/var/lib/kubelet/pods/f58e1b66-6f12-4a7b-93d4-02c43e499f7a/volumes" Nov 01 22:47:21.590598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:21.590559 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-7sgls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:47:21.590766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:21.590619 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:47:21.638496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:21.638458 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:47:21.638676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:21.638540 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:47:22.605830 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:22.605795 6287 patch_prober.go:29] interesting pod/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:47:22.606225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:22.605850 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:47:24.448350 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:24.448318 6287 patch_prober.go:29] interesting pod/e2e-worker-w1-381659-openshift-default-894db6755-j5226 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:47:24.448864 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:24.448376 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:47:25.621797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.621767 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:47:25.671749 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.671726 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:47:25.705990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.705963 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-data\") pod \"de6f8907-e8c1-493e-9a47-6aaa669e3796\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " Nov 01 22:47:25.706161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.705999 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-socket\") pod \"de6f8907-e8c1-493e-9a47-6aaa669e3796\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " Nov 01 22:47:25.706273 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706165 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-podinfo\") pod \"de6f8907-e8c1-493e-9a47-6aaa669e3796\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " Nov 01 22:47:25.706273 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706196 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-envoy\") pod \"de6f8907-e8c1-493e-9a47-6aaa669e3796\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " Nov 01 22:47:25.706389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706362 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-data" pod "de6f8907-e8c1-493e-9a47-6aaa669e3796" (UID: "de6f8907-e8c1-493e-9a47-6aaa669e3796"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.706389 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706371 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-credential-socket\") pod \"de6f8907-e8c1-493e-9a47-6aaa669e3796\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " Nov 01 22:47:25.706468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706382 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-socket" pod "de6f8907-e8c1-493e-9a47-6aaa669e3796" (UID: "de6f8907-e8c1-493e-9a47-6aaa669e3796"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.706468 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706453 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-token\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-token\") pod \"de6f8907-e8c1-493e-9a47-6aaa669e3796\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " Nov 01 22:47:25.706571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706482 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de6f8907-e8c1-493e-9a47-6aaa669e3796-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6f8907-e8c1-493e-9a47-6aaa669e3796-istiod-ca-cert\") pod \"de6f8907-e8c1-493e-9a47-6aaa669e3796\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " Nov 01 22:47:25.706571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706554 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-certs\") pod \"de6f8907-e8c1-493e-9a47-6aaa669e3796\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " Nov 01 22:47:25.706674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706570 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-credential-socket" pod "de6f8907-e8c1-493e-9a47-6aaa669e3796" (UID: "de6f8907-e8c1-493e-9a47-6aaa669e3796"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.706674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706606 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-kube-api-access-2t9r5\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-kube-api-access-2t9r5\") pod \"de6f8907-e8c1-493e-9a47-6aaa669e3796\" (UID: \"de6f8907-e8c1-493e-9a47-6aaa669e3796\") " Nov 01 22:47:25.706939 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.706919 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6f8907-e8c1-493e-9a47-6aaa669e3796-istiod-ca-cert" pod "de6f8907-e8c1-493e-9a47-6aaa669e3796" (UID: "de6f8907-e8c1-493e-9a47-6aaa669e3796"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:25.710781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.708528 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.710781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.708560 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.710781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.708587 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.710781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.708619 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6f8907-e8c1-493e-9a47-6aaa669e3796-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.710781 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.709263 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-token" pod "de6f8907-e8c1-493e-9a47-6aaa669e3796" (UID: "de6f8907-e8c1-493e-9a47-6aaa669e3796"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:25.712714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.711107 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-kube-api-access-2t9r5" pod "de6f8907-e8c1-493e-9a47-6aaa669e3796" (UID: "de6f8907-e8c1-493e-9a47-6aaa669e3796"). InnerVolumeSpecName "kube-api-access-2t9r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:25.712714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.712356 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-certs" pod "de6f8907-e8c1-493e-9a47-6aaa669e3796" (UID: "de6f8907-e8c1-493e-9a47-6aaa669e3796"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.712714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.712402 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-envoy" pod "de6f8907-e8c1-493e-9a47-6aaa669e3796" (UID: "de6f8907-e8c1-493e-9a47-6aaa669e3796"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.719547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.719511 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-podinfo" pod "de6f8907-e8c1-493e-9a47-6aaa669e3796" (UID: "de6f8907-e8c1-493e-9a47-6aaa669e3796"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:25.809865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.809832 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-podinfo\") pod \"f6383f0b-7182-4aae-80f8-0bb64da62924\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " Nov 01 22:47:25.810043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.809944 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-token\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-token\") pod \"f6383f0b-7182-4aae-80f8-0bb64da62924\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " Nov 01 22:47:25.810043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.809969 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6383f0b-7182-4aae-80f8-0bb64da62924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6383f0b-7182-4aae-80f8-0bb64da62924-istiod-ca-cert\") pod \"f6383f0b-7182-4aae-80f8-0bb64da62924\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " Nov 01 22:47:25.810043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.809993 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-kube-api-access-kqr4k\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-kube-api-access-kqr4k\") pod \"f6383f0b-7182-4aae-80f8-0bb64da62924\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " Nov 01 22:47:25.810043 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.810016 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-envoy\") pod \"f6383f0b-7182-4aae-80f8-0bb64da62924\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " Nov 01 22:47:25.810317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.810057 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-credential-socket\") pod \"f6383f0b-7182-4aae-80f8-0bb64da62924\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " Nov 01 22:47:25.810500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.810475 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-certs\") pod \"f6383f0b-7182-4aae-80f8-0bb64da62924\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " Nov 01 22:47:25.810704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.810552 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-credential-socket" pod "f6383f0b-7182-4aae-80f8-0bb64da62924" (UID: "f6383f0b-7182-4aae-80f8-0bb64da62924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.810704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.810581 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6383f0b-7182-4aae-80f8-0bb64da62924-istiod-ca-cert" pod "f6383f0b-7182-4aae-80f8-0bb64da62924" (UID: "f6383f0b-7182-4aae-80f8-0bb64da62924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:25.810851 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.810795 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-certs" pod "f6383f0b-7182-4aae-80f8-0bb64da62924" (UID: "f6383f0b-7182-4aae-80f8-0bb64da62924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.810991 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.810964 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-socket\") pod \"f6383f0b-7182-4aae-80f8-0bb64da62924\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " Nov 01 22:47:25.811292 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.811259 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-socket" pod "f6383f0b-7182-4aae-80f8-0bb64da62924" (UID: "f6383f0b-7182-4aae-80f8-0bb64da62924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.811404 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.811377 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-data\") pod \"f6383f0b-7182-4aae-80f8-0bb64da62924\" (UID: \"f6383f0b-7182-4aae-80f8-0bb64da62924\") " Nov 01 22:47:25.811636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.811616 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-data" pod "f6383f0b-7182-4aae-80f8-0bb64da62924" (UID: "f6383f0b-7182-4aae-80f8-0bb64da62924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.812247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812224 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.812376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812251 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9r5\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-kube-api-access-2t9r5\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.812376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812265 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6383f0b-7182-4aae-80f8-0bb64da62924-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.812376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812276 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.812376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812285 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.812376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812294 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.812376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812304 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.812376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812315 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.812376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812327 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.812376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812341 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6f8907-e8c1-493e-9a47-6aaa669e3796-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.813004 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812979 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-podinfo" pod "f6383f0b-7182-4aae-80f8-0bb64da62924" (UID: "f6383f0b-7182-4aae-80f8-0bb64da62924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:25.813158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.812994 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-token" pod "f6383f0b-7182-4aae-80f8-0bb64da62924" (UID: "f6383f0b-7182-4aae-80f8-0bb64da62924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:25.813158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.813067 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-envoy" pod "f6383f0b-7182-4aae-80f8-0bb64da62924" (UID: "f6383f0b-7182-4aae-80f8-0bb64da62924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:25.813158 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.813125 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-kube-api-access-kqr4k" pod "f6383f0b-7182-4aae-80f8-0bb64da62924" (UID: "f6383f0b-7182-4aae-80f8-0bb64da62924"). InnerVolumeSpecName "kube-api-access-kqr4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:25.913695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.913617 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.913695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.913645 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqr4k\" (UniqueName: \"kubernetes.io/projected/f6383f0b-7182-4aae-80f8-0bb64da62924-kube-api-access-kqr4k\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.913695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.913657 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.913695 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.913665 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6383f0b-7182-4aae-80f8-0bb64da62924-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:25.968253 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.968219 6287 generic.go:356] "Generic (PLEG): container finished" podID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerID="2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184" exitCode=0 Nov 01 22:47:25.968437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.968295 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" Nov 01 22:47:25.968437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.968300 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" event={"ID":"de6f8907-e8c1-493e-9a47-6aaa669e3796","Type":"ContainerDied","Data":"2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184"} Nov 01 22:47:25.968437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.968380 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls" event={"ID":"de6f8907-e8c1-493e-9a47-6aaa669e3796","Type":"ContainerDied","Data":"afd6f351d44f8082a683cb94f785a25b67dd2e978f13cbf7ae021209d56cdf0a"} Nov 01 22:47:25.968437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.968407 6287 scope.go:122] "RemoveContainer" containerID="2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184" Nov 01 22:47:25.969981 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.969956 6287 generic.go:356] "Generic (PLEG): container finished" podID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerID="a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490" exitCode=0 Nov 01 22:47:25.970066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.970007 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" Nov 01 22:47:25.970066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.970041 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" event={"ID":"f6383f0b-7182-4aae-80f8-0bb64da62924","Type":"ContainerDied","Data":"a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490"} Nov 01 22:47:25.970193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.970093 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226" event={"ID":"f6383f0b-7182-4aae-80f8-0bb64da62924","Type":"ContainerDied","Data":"6d0fc536a4740c0e2417049d384bb8320aa7279c3cfc6f17442cd3ab0bb4c57e"} Nov 01 22:47:25.982008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.981990 6287 scope.go:122] "RemoveContainer" containerID="2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184" Nov 01 22:47:25.982350 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:25.982326 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184\": container with ID starting with 2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184 not found: ID does not exist" containerID="2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184" Nov 01 22:47:25.982411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.982363 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184"} err="failed to get container status \"2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184\": rpc error: code = NotFound desc = could not find container \"2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184\": container with ID starting with 2bf9ac3ac26b6c7b70cddf2de6f49c9cca150f98d54327b5c8f48a7da285a184 not found: ID does not exist" Nov 01 22:47:25.982411 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:25.982385 6287 scope.go:122] "RemoveContainer" containerID="a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490" Nov 01 22:47:26.031174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.026869 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls"] Nov 01 22:47:26.032294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.031780 6287 scope.go:122] "RemoveContainer" containerID="a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490" Nov 01 22:47:26.032294 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:26.032225 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490\": container with ID starting with a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490 not found: ID does not exist" containerID="a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490" Nov 01 22:47:26.032294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.032265 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490"} err="failed to get container status \"a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490\": rpc error: code = NotFound desc = could not find container \"a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490\": container with ID starting with a1b6ad96de46a8be346698990be5a2a060d922c5cdf68bb1c2a52148d2943490 not found: ID does not exist" Nov 01 22:47:26.034534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.034504 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-7sgls"] Nov 01 22:47:26.040553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.040520 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226"] Nov 01 22:47:26.044130 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.044092 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-381659-openshift-default-894db6755-j5226"] Nov 01 22:47:26.523169 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.523086 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:47:26.565075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.565047 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:47:26.665596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.662094 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-socket\") pod \"03620ee4-ac0f-435c-a4d3-011c85212538\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666284 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-credential-socket\") pod \"fa05e16a-6810-4247-8326-231181aa3987\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666359 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-istio-token\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-istio-token\") pod \"03620ee4-ac0f-435c-a4d3-011c85212538\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666395 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-data\") pod \"03620ee4-ac0f-435c-a4d3-011c85212538\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666425 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-credential-socket\") pod \"03620ee4-ac0f-435c-a4d3-011c85212538\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666463 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-certs\") pod \"03620ee4-ac0f-435c-a4d3-011c85212538\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666493 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03620ee4-ac0f-435c-a4d3-011c85212538-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03620ee4-ac0f-435c-a4d3-011c85212538-istio-podinfo\") pod \"03620ee4-ac0f-435c-a4d3-011c85212538\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666535 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa05e16a-6810-4247-8326-231181aa3987-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05e16a-6810-4247-8326-231181aa3987-istiod-ca-cert\") pod \"fa05e16a-6810-4247-8326-231181aa3987\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666569 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-kube-api-access-hvzrj\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-kube-api-access-hvzrj\") pod \"03620ee4-ac0f-435c-a4d3-011c85212538\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666596 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-socket\") pod \"fa05e16a-6810-4247-8326-231181aa3987\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666624 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa05e16a-6810-4247-8326-231181aa3987-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05e16a-6810-4247-8326-231181aa3987-istio-podinfo\") pod \"fa05e16a-6810-4247-8326-231181aa3987\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666674 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-certs\") pod \"fa05e16a-6810-4247-8326-231181aa3987\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666712 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03620ee4-ac0f-435c-a4d3-011c85212538-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03620ee4-ac0f-435c-a4d3-011c85212538-istiod-ca-cert\") pod \"03620ee4-ac0f-435c-a4d3-011c85212538\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666768 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-kube-api-access-q6b2d\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-kube-api-access-q6b2d\") pod \"fa05e16a-6810-4247-8326-231181aa3987\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666814 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-istio-token\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-istio-token\") pod \"fa05e16a-6810-4247-8326-231181aa3987\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666851 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-envoy\") pod \"fa05e16a-6810-4247-8326-231181aa3987\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666919 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-data\") pod \"fa05e16a-6810-4247-8326-231181aa3987\" (UID: \"fa05e16a-6810-4247-8326-231181aa3987\") " Nov 01 22:47:26.667381 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.666947 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-envoy\") pod \"03620ee4-ac0f-435c-a4d3-011c85212538\" (UID: \"03620ee4-ac0f-435c-a4d3-011c85212538\") " Nov 01 22:47:26.668843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.668657 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-socket" pod "03620ee4-ac0f-435c-a4d3-011c85212538" (UID: "03620ee4-ac0f-435c-a4d3-011c85212538"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.670132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.669459 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.670500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.670290 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-socket" pod "fa05e16a-6810-4247-8326-231181aa3987" (UID: "fa05e16a-6810-4247-8326-231181aa3987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.670935 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.670635 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-credential-socket" pod "fa05e16a-6810-4247-8326-231181aa3987" (UID: "fa05e16a-6810-4247-8326-231181aa3987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.673075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.672156 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-data" pod "03620ee4-ac0f-435c-a4d3-011c85212538" (UID: "03620ee4-ac0f-435c-a4d3-011c85212538"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.673075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.672655 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-credential-socket" pod "03620ee4-ac0f-435c-a4d3-011c85212538" (UID: "03620ee4-ac0f-435c-a4d3-011c85212538"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.673075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.672830 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa05e16a-6810-4247-8326-231181aa3987-istiod-ca-cert" pod "fa05e16a-6810-4247-8326-231181aa3987" (UID: "fa05e16a-6810-4247-8326-231181aa3987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:26.673613 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.673590 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-certs" pod "03620ee4-ac0f-435c-a4d3-011c85212538" (UID: "03620ee4-ac0f-435c-a4d3-011c85212538"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.677598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.677487 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-istio-token" pod "03620ee4-ac0f-435c-a4d3-011c85212538" (UID: "03620ee4-ac0f-435c-a4d3-011c85212538"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:26.677930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.677886 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-certs" pod "fa05e16a-6810-4247-8326-231181aa3987" (UID: "fa05e16a-6810-4247-8326-231181aa3987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.678038 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.678017 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-data" pod "fa05e16a-6810-4247-8326-231181aa3987" (UID: "fa05e16a-6810-4247-8326-231181aa3987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.678124 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.677967 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03620ee4-ac0f-435c-a4d3-011c85212538-istiod-ca-cert" pod "03620ee4-ac0f-435c-a4d3-011c85212538" (UID: "03620ee4-ac0f-435c-a4d3-011c85212538"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:26.679399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.679322 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-kube-api-access-q6b2d" pod "fa05e16a-6810-4247-8326-231181aa3987" (UID: "fa05e16a-6810-4247-8326-231181aa3987"). InnerVolumeSpecName "kube-api-access-q6b2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:26.679399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.679360 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03620ee4-ac0f-435c-a4d3-011c85212538-istio-podinfo" pod "03620ee4-ac0f-435c-a4d3-011c85212538" (UID: "03620ee4-ac0f-435c-a4d3-011c85212538"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:26.681423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.681403 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-envoy" pod "03620ee4-ac0f-435c-a4d3-011c85212538" (UID: "03620ee4-ac0f-435c-a4d3-011c85212538"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.681853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.681818 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa05e16a-6810-4247-8326-231181aa3987-istio-podinfo" pod "fa05e16a-6810-4247-8326-231181aa3987" (UID: "fa05e16a-6810-4247-8326-231181aa3987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:26.682581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.682300 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-kube-api-access-hvzrj" pod "03620ee4-ac0f-435c-a4d3-011c85212538" (UID: "03620ee4-ac0f-435c-a4d3-011c85212538"). InnerVolumeSpecName "kube-api-access-hvzrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:26.682581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.682430 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-istio-token" pod "fa05e16a-6810-4247-8326-231181aa3987" (UID: "fa05e16a-6810-4247-8326-231181aa3987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:26.682581 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.682539 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-envoy" pod "fa05e16a-6810-4247-8326-231181aa3987" (UID: "fa05e16a-6810-4247-8326-231181aa3987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:26.770443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770397 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770438 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770453 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770469 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03620ee4-ac0f-435c-a4d3-011c85212538-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770483 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa05e16a-6810-4247-8326-231181aa3987-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770499 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzrj\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-kube-api-access-hvzrj\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770512 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770526 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa05e16a-6810-4247-8326-231181aa3987-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770542 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770554 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03620ee4-ac0f-435c-a4d3-011c85212538-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770569 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6b2d\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-kube-api-access-q6b2d\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770582 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa05e16a-6810-4247-8326-231181aa3987-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770597 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770610 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770623 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03620ee4-ac0f-435c-a4d3-011c85212538-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770636 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa05e16a-6810-4247-8326-231181aa3987-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.770651 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03620ee4-ac0f-435c-a4d3-011c85212538-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:26.981168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.981083 6287 generic.go:356] "Generic (PLEG): container finished" podID="fa05e16a-6810-4247-8326-231181aa3987" containerID="546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075" exitCode=0 Nov 01 22:47:26.981168 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.981161 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" Nov 01 22:47:26.981369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.981172 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" event={"ID":"fa05e16a-6810-4247-8326-231181aa3987","Type":"ContainerDied","Data":"546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075"} Nov 01 22:47:26.981369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.981233 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr" event={"ID":"fa05e16a-6810-4247-8326-231181aa3987","Type":"ContainerDied","Data":"a369740fbf234b02ddc61552894ece6dae52db5d7321d6dedaa09143db29e8f5"} Nov 01 22:47:26.981369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.981263 6287 scope.go:122] "RemoveContainer" containerID="546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075" Nov 01 22:47:26.982934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.982907 6287 generic.go:356] "Generic (PLEG): container finished" podID="03620ee4-ac0f-435c-a4d3-011c85212538" containerID="f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd" exitCode=0 Nov 01 22:47:26.983054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.982966 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" Nov 01 22:47:26.983054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.982998 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" event={"ID":"03620ee4-ac0f-435c-a4d3-011c85212538","Type":"ContainerDied","Data":"f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd"} Nov 01 22:47:26.983218 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.983063 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg" event={"ID":"03620ee4-ac0f-435c-a4d3-011c85212538","Type":"ContainerDied","Data":"d7301ed3bafe18dca00305f2bcbf77fd692e8d75fe7d1468d6ad08b0ce10e0a6"} Nov 01 22:47:26.995214 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.995196 6287 scope.go:122] "RemoveContainer" containerID="546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075" Nov 01 22:47:26.995502 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:26.995484 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075\": container with ID starting with 546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075 not found: ID does not exist" containerID="546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075" Nov 01 22:47:26.995557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.995509 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075"} err="failed to get container status \"546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075\": rpc error: code = NotFound desc = could not find container \"546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075\": container with ID starting with 546dc614a8806cff466911c9e62c1de8a2cc9e5bb7f24957e076edf3b4cc5075 not found: ID does not exist" Nov 01 22:47:26.995557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:26.995525 6287 scope.go:122] "RemoveContainer" containerID="f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd" Nov 01 22:47:27.006918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.006895 6287 scope.go:122] "RemoveContainer" containerID="f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd" Nov 01 22:47:27.007212 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:27.007194 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd\": container with ID starting with f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd not found: ID does not exist" containerID="f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd" Nov 01 22:47:27.007274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.007219 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd"} err="failed to get container status \"f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd\": rpc error: code = NotFound desc = could not find container \"f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd\": container with ID starting with f484cc0f8efd68521e0e92c57c0f73339ef5addce7042b48094f412924307cdd not found: ID does not exist" Nov 01 22:47:27.059696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.059655 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr"] Nov 01 22:47:27.064162 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.063531 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-hbxzr"] Nov 01 22:47:27.077234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.077186 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg"] Nov 01 22:47:27.087279 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.087250 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3d8490-openshift-default-8474f778b5-rfrtg"] Nov 01 22:47:27.472663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.472633 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" path="/var/lib/kubelet/pods/03620ee4-ac0f-435c-a4d3-011c85212538/volumes" Nov 01 22:47:27.473267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.473234 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" path="/var/lib/kubelet/pods/de6f8907-e8c1-493e-9a47-6aaa669e3796/volumes" Nov 01 22:47:27.473947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.473923 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" path="/var/lib/kubelet/pods/f6383f0b-7182-4aae-80f8-0bb64da62924/volumes" Nov 01 22:47:27.474789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:27.474772 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa05e16a-6810-4247-8326-231181aa3987" path="/var/lib/kubelet/pods/fa05e16a-6810-4247-8326-231181aa3987/volumes" Nov 01 22:47:28.686149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686105 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s"] Nov 01 22:47:28.686662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686588 6287 cpu_manager.go:414] "containerMap: removing container" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" Nov 01 22:47:28.686662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686606 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" Nov 01 22:47:28.686662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686623 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f58e1b66-6f12-4a7b-93d4-02c43e499f7a" containerName="payload-processing" Nov 01 22:47:28.686662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686632 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58e1b66-6f12-4a7b-93d4-02c43e499f7a" containerName="payload-processing" Nov 01 22:47:28.686662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686658 6287 cpu_manager.go:414] "containerMap: removing container" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" Nov 01 22:47:28.686662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686664 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686679 6287 cpu_manager.go:414] "containerMap: removing container" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686683 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686694 6287 cpu_manager.go:414] "containerMap: removing container" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686702 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686717 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686724 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686735 6287 cpu_manager.go:414] "containerMap: removing container" podUID="75e8b716-bb96-4a9b-bc0d-43f1fddb02ed" containerName="revoke-keys" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686743 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e8b716-bb96-4a9b-bc0d-43f1fddb02ed" containerName="revoke-keys" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686834 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6f8907-e8c1-493e-9a47-6aaa669e3796" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686846 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="03620ee4-ac0f-435c-a4d3-011c85212538" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686853 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6383f0b-7182-4aae-80f8-0bb64da62924" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686858 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e8b716-bb96-4a9b-bc0d-43f1fddb02ed" containerName="revoke-keys" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686864 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="667699ea-ed8e-4055-ae52-3bd87e5e966d" containerName="istio-proxy" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686872 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58e1b66-6f12-4a7b-93d4-02c43e499f7a" containerName="payload-processing" Nov 01 22:47:28.686971 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.686888 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa05e16a-6810-4247-8326-231181aa3987" containerName="istio-proxy" Nov 01 22:47:28.692758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.692731 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:28.695216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.695189 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66"] Nov 01 22:47:28.703186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.702003 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.703681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.703656 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:47:28.703853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.703692 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s"] Nov 01 22:47:28.703853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.703803 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:47:28.703853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.703813 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:47:28.710869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.710087 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:47:28.713222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.713200 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66"] Nov 01 22:47:28.796473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.796444 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b706d81-5mk66\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.796652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.796493 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b132126-nkn2s\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:28.796652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.796516 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b132126-nkn2s\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:28.796652 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.796607 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b706d81-5mk66\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.796824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.796671 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtkn\" (UniqueName: \"kubernetes.io/projected/e2dd6b25-84e7-4518-b112-3ad691d96e98-kube-api-access-nrtkn\") pod \"maas-api-revoke-sub-4b706d81-5mk66\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.796824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.796758 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6nc\" (UniqueName: \"kubernetes.io/projected/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-kube-api-access-zk6nc\") pod \"maas-api-revoke-sub-7b132126-nkn2s\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:28.898085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.898054 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtkn\" (UniqueName: \"kubernetes.io/projected/e2dd6b25-84e7-4518-b112-3ad691d96e98-kube-api-access-nrtkn\") pod \"maas-api-revoke-sub-4b706d81-5mk66\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.898263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.898164 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6nc\" (UniqueName: \"kubernetes.io/projected/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-kube-api-access-zk6nc\") pod \"maas-api-revoke-sub-7b132126-nkn2s\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:28.898263 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.898219 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b706d81-5mk66\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.898369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.898266 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b132126-nkn2s\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:28.898369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.898290 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b132126-nkn2s\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:28.898369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.898311 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b706d81-5mk66\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.898925 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.898892 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b132126-nkn2s\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:28.899067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.898994 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b706d81-5mk66\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.900657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.900628 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b706d81-5mk66\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.900760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.900654 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b132126-nkn2s\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:28.906485 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.906463 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtkn\" (UniqueName: \"kubernetes.io/projected/e2dd6b25-84e7-4518-b112-3ad691d96e98-kube-api-access-nrtkn\") pod \"maas-api-revoke-sub-4b706d81-5mk66\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:28.906682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:28.906658 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6nc\" (UniqueName: \"kubernetes.io/projected/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-kube-api-access-zk6nc\") pod \"maas-api-revoke-sub-7b132126-nkn2s\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:29.018728 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:29.018700 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:29.028017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:29.027990 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:29.326204 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:29.326175 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s"] Nov 01 22:47:29.326587 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:47:29.326565 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab9a3cf_0ee0_4cbc_9a55_0f1c81996fb6.slice/crio-809ea4b25f66ebf978ca7d5ff2e8091fbb59418f4d15adecdb7c830d5552cf97 WatchSource:0}: Error finding container 809ea4b25f66ebf978ca7d5ff2e8091fbb59418f4d15adecdb7c830d5552cf97: Status 404 returned error can't find the container with id 809ea4b25f66ebf978ca7d5ff2e8091fbb59418f4d15adecdb7c830d5552cf97 Nov 01 22:47:29.385054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:29.385027 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66"] Nov 01 22:47:29.385285 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:47:29.385263 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dd6b25_84e7_4518_b112_3ad691d96e98.slice/crio-1dc8e29743777e12c64cb9ba5fe694613647628871d3d38403b771f6fd88063a WatchSource:0}: Error finding container 1dc8e29743777e12c64cb9ba5fe694613647628871d3d38403b771f6fd88063a: Status 404 returned error can't find the container with id 1dc8e29743777e12c64cb9ba5fe694613647628871d3d38403b771f6fd88063a Nov 01 22:47:30.024238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:30.024200 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" event={"ID":"e2dd6b25-84e7-4518-b112-3ad691d96e98","Type":"ContainerStarted","Data":"293071a92c5c5d68baa904d7dc1e58b09049ffa09ec2ea6485a377e2eae5ee47"} Nov 01 22:47:30.024730 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:30.024272 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" event={"ID":"e2dd6b25-84e7-4518-b112-3ad691d96e98","Type":"ContainerStarted","Data":"1dc8e29743777e12c64cb9ba5fe694613647628871d3d38403b771f6fd88063a"} Nov 01 22:47:30.026398 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:30.026349 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" event={"ID":"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6","Type":"ContainerStarted","Data":"e14bd6e5ae8e6694600d8d2bafa4ca0af247bdafebbe2ae3c829e2cd56a97fc0"} Nov 01 22:47:30.026657 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:30.026403 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" event={"ID":"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6","Type":"ContainerStarted","Data":"809ea4b25f66ebf978ca7d5ff2e8091fbb59418f4d15adecdb7c830d5552cf97"} Nov 01 22:47:30.051126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:30.051066 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" podStartSLOduration=2.051052915 podStartE2EDuration="2.051052915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:30.047299497 +0000 UTC m=+1462.695798732" watchObservedRunningTime="2026-11-01 22:47:30.051052915 +0000 UTC m=+1462.699552151" Nov 01 22:47:30.073019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:30.072973 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" podStartSLOduration=2.072960246 podStartE2EDuration="2.072960246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:30.070675386 +0000 UTC m=+1462.719174624" watchObservedRunningTime="2026-11-01 22:47:30.072960246 +0000 UTC m=+1462.721459481" Nov 01 22:47:30.710251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:30.710213 6287 scope.go:122] "RemoveContainer" containerID="b8ea87abc55f42f571514fbbe780988b54d49c49ce80a80135852edf0d2e458c" Nov 01 22:47:30.721475 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:30.721457 6287 scope.go:122] "RemoveContainer" containerID="f861ebe0bacd78cf80771f9546e5d448c89090042d5d7848870891558aa666ac" Nov 01 22:47:30.731230 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:30.731211 6287 scope.go:122] "RemoveContainer" containerID="2d594fd5dc3152c73073b43ed997fd9e57ffcde650ff1e91799733690ff04b9c" Nov 01 22:47:31.035268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:31.035238 6287 generic.go:356] "Generic (PLEG): container finished" podID="e2dd6b25-84e7-4518-b112-3ad691d96e98" containerID="293071a92c5c5d68baa904d7dc1e58b09049ffa09ec2ea6485a377e2eae5ee47" exitCode=0 Nov 01 22:47:31.035780 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:31.035335 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" event={"ID":"e2dd6b25-84e7-4518-b112-3ad691d96e98","Type":"ContainerDied","Data":"293071a92c5c5d68baa904d7dc1e58b09049ffa09ec2ea6485a377e2eae5ee47"} Nov 01 22:47:31.037047 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:31.037024 6287 generic.go:356] "Generic (PLEG): container finished" podID="bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6" containerID="e14bd6e5ae8e6694600d8d2bafa4ca0af247bdafebbe2ae3c829e2cd56a97fc0" exitCode=0 Nov 01 22:47:31.037208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:31.037101 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" event={"ID":"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6","Type":"ContainerDied","Data":"e14bd6e5ae8e6694600d8d2bafa4ca0af247bdafebbe2ae3c829e2cd56a97fc0"} Nov 01 22:47:32.193918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.193889 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:32.235705 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.235678 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-openshift-ca\") pod \"e2dd6b25-84e7-4518-b112-3ad691d96e98\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " Nov 01 22:47:32.235853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.235743 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2dd6b25-84e7-4518-b112-3ad691d96e98-kube-api-access-nrtkn\" (UniqueName: \"kubernetes.io/projected/e2dd6b25-84e7-4518-b112-3ad691d96e98-kube-api-access-nrtkn\") pod \"e2dd6b25-84e7-4518-b112-3ad691d96e98\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " Nov 01 22:47:32.235853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.235846 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-xks-ca\") pod \"e2dd6b25-84e7-4518-b112-3ad691d96e98\" (UID: \"e2dd6b25-84e7-4518-b112-3ad691d96e98\") " Nov 01 22:47:32.236059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.236032 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-openshift-ca" pod "e2dd6b25-84e7-4518-b112-3ad691d96e98" (UID: "e2dd6b25-84e7-4518-b112-3ad691d96e98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:32.236435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.236413 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:32.237954 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.237929 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dd6b25-84e7-4518-b112-3ad691d96e98-kube-api-access-nrtkn" pod "e2dd6b25-84e7-4518-b112-3ad691d96e98" (UID: "e2dd6b25-84e7-4518-b112-3ad691d96e98"). InnerVolumeSpecName "kube-api-access-nrtkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:32.238032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.238008 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-xks-ca" pod "e2dd6b25-84e7-4518-b112-3ad691d96e98" (UID: "e2dd6b25-84e7-4518-b112-3ad691d96e98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:32.239777 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.239755 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:32.336918 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.336835 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-openshift-ca\") pod \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " Nov 01 22:47:32.337096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.336920 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-xks-ca\") pod \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " Nov 01 22:47:32.337096 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.336962 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-kube-api-access-zk6nc\" (UniqueName: \"kubernetes.io/projected/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-kube-api-access-zk6nc\") pod \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\" (UID: \"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6\") " Nov 01 22:47:32.337268 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.337240 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-openshift-ca" pod "bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6" (UID: "bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:32.337330 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.337285 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtkn\" (UniqueName: \"kubernetes.io/projected/e2dd6b25-84e7-4518-b112-3ad691d96e98-kube-api-access-nrtkn\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:32.337330 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.337301 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2dd6b25-84e7-4518-b112-3ad691d96e98-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:32.339137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.339098 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-kube-api-access-zk6nc" pod "bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6" (UID: "bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6"). InnerVolumeSpecName "kube-api-access-zk6nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:32.339259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.339172 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-xks-ca" pod "bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6" (UID: "bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:32.438619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.438587 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk6nc\" (UniqueName: \"kubernetes.io/projected/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-kube-api-access-zk6nc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:32.438619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.438617 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:32.438803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:32.438631 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:33.052902 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:33.052876 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" Nov 01 22:47:33.053161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:33.052868 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s" event={"ID":"bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6","Type":"ContainerDied","Data":"809ea4b25f66ebf978ca7d5ff2e8091fbb59418f4d15adecdb7c830d5552cf97"} Nov 01 22:47:33.053161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:33.053032 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="809ea4b25f66ebf978ca7d5ff2e8091fbb59418f4d15adecdb7c830d5552cf97" Nov 01 22:47:33.054386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:33.054368 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" Nov 01 22:47:33.054492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:33.054395 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66" event={"ID":"e2dd6b25-84e7-4518-b112-3ad691d96e98","Type":"ContainerDied","Data":"1dc8e29743777e12c64cb9ba5fe694613647628871d3d38403b771f6fd88063a"} Nov 01 22:47:33.054492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:33.054438 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dc8e29743777e12c64cb9ba5fe694613647628871d3d38403b771f6fd88063a" Nov 01 22:47:34.955728 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:34.955687 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" containerID="cri-o://23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e" gracePeriod=45 Nov 01 22:47:35.230863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.230835 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:47:35.268661 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.268626 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b2092d-b96e-485d-922d-3322cafbcf4d-kube-api-access-w5z6h\" (UniqueName: \"kubernetes.io/projected/14b2092d-b96e-485d-922d-3322cafbcf4d-kube-api-access-w5z6h\") pod \"14b2092d-b96e-485d-922d-3322cafbcf4d\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " Nov 01 22:47:35.268838 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.268820 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-tmp-dir\") pod \"14b2092d-b96e-485d-922d-3322cafbcf4d\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " Nov 01 22:47:35.268917 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.268874 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14b2092d-b96e-485d-922d-3322cafbcf4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/14b2092d-b96e-485d-922d-3322cafbcf4d-tls-certs\") pod \"14b2092d-b96e-485d-922d-3322cafbcf4d\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " Nov 01 22:47:35.268983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.268921 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-dshm\") pod \"14b2092d-b96e-485d-922d-3322cafbcf4d\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " Nov 01 22:47:35.268983 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.268966 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-home\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-home\") pod \"14b2092d-b96e-485d-922d-3322cafbcf4d\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " Nov 01 22:47:35.269093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.269046 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-model-cache\") pod \"14b2092d-b96e-485d-922d-3322cafbcf4d\" (UID: \"14b2092d-b96e-485d-922d-3322cafbcf4d\") " Nov 01 22:47:35.269235 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.269204 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-tmp-dir" pod "14b2092d-b96e-485d-922d-3322cafbcf4d" (UID: "14b2092d-b96e-485d-922d-3322cafbcf4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:35.269402 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.269374 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-home" pod "14b2092d-b96e-485d-922d-3322cafbcf4d" (UID: "14b2092d-b96e-485d-922d-3322cafbcf4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:35.269632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.269581 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.269632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.269448 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-model-cache" pod "14b2092d-b96e-485d-922d-3322cafbcf4d" (UID: "14b2092d-b96e-485d-922d-3322cafbcf4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:35.272385 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.272349 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b2092d-b96e-485d-922d-3322cafbcf4d-kube-api-access-w5z6h" pod "14b2092d-b96e-485d-922d-3322cafbcf4d" (UID: "14b2092d-b96e-485d-922d-3322cafbcf4d"). InnerVolumeSpecName "kube-api-access-w5z6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:35.272495 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.272386 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b2092d-b96e-485d-922d-3322cafbcf4d-tls-certs" pod "14b2092d-b96e-485d-922d-3322cafbcf4d" (UID: "14b2092d-b96e-485d-922d-3322cafbcf4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:35.272885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.272618 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-dshm" pod "14b2092d-b96e-485d-922d-3322cafbcf4d" (UID: "14b2092d-b96e-485d-922d-3322cafbcf4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:35.370774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.370741 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b2092d-b96e-485d-922d-3322cafbcf4d-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.370774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.370770 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.370774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.370778 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.371014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.370786 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b2092d-b96e-485d-922d-3322cafbcf4d-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:35.371014 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:35.370799 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5z6h\" (UniqueName: \"kubernetes.io/projected/14b2092d-b96e-485d-922d-3322cafbcf4d-kube-api-access-w5z6h\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:36.088913 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:36.088803 6287 generic.go:356] "Generic (PLEG): container finished" podID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerID="23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e" exitCode=0 Nov 01 22:47:36.089690 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:36.088885 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" event={"ID":"14b2092d-b96e-485d-922d-3322cafbcf4d","Type":"ContainerDied","Data":"23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e"} Nov 01 22:47:36.089927 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:36.089878 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" event={"ID":"14b2092d-b96e-485d-922d-3322cafbcf4d","Type":"ContainerDied","Data":"69addb43f7be5325449aba35344136bc7142ad5df778df2aa207a742a0b6eb48"} Nov 01 22:47:36.090133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:36.090057 6287 scope.go:122] "RemoveContainer" containerID="23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e" Nov 01 22:47:36.090493 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:36.090440 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl" Nov 01 22:47:36.120869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:36.120837 6287 scope.go:122] "RemoveContainer" containerID="23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e" Nov 01 22:47:36.121177 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:36.121156 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e\": container with ID starting with 23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e not found: ID does not exist" containerID="23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e" Nov 01 22:47:36.121236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:36.121185 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e"} err="failed to get container status \"23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e\": rpc error: code = NotFound desc = could not find container \"23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e\": container with ID starting with 23f3db7dd980c7a8bfce74b8505a7dbe0aa7a6a347806bd34b3934f26a49082e not found: ID does not exist" Nov 01 22:47:36.135872 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:36.135844 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl"] Nov 01 22:47:36.147004 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:36.146979 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b28a40/distinct-w6-b28a40-kserve-866497bdf6-xmtsl"] Nov 01 22:47:37.473302 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:37.471746 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" path="/var/lib/kubelet/pods/14b2092d-b96e-485d-922d-3322cafbcf4d/volumes" Nov 01 22:47:45.588188 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:47:45.588148 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:47:48.911157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:48.911089 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458"] Nov 01 22:47:48.916310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:48.916282 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-b28a40-hg458"] Nov 01 22:47:49.471065 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:49.471027 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e8b716-bb96-4a9b-bc0d-43f1fddb02ed" path="/var/lib/kubelet/pods/75e8b716-bb96-4a9b-bc0d-43f1fddb02ed/volumes" Nov 01 22:47:53.174478 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:53.174447 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk"] Nov 01 22:47:53.174922 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:53.174655 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" containerID="cri-o://dd42c70b0feda16a153cb16450acbf5358dba85c444c04328685b180c06e07ae" gracePeriod=30 Nov 01 22:47:57.130066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:57.130028 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz"] Nov 01 22:47:57.130590 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:57.130307 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" containerID="cri-o://3c686ccbc58dfd3b8af485b6c6484bf3a88152ae37de7c71d8bd0cee76703bc6" gracePeriod=30 Nov 01 22:47:58.305960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.305925 6287 generic.go:356] "Generic (PLEG): container finished" podID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerID="dd42c70b0feda16a153cb16450acbf5358dba85c444c04328685b180c06e07ae" exitCode=0 Nov 01 22:47:58.306334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.305993 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" event={"ID":"a234993f-449e-4d56-82c5-8bc2282a67fd","Type":"ContainerDied","Data":"dd42c70b0feda16a153cb16450acbf5358dba85c444c04328685b180c06e07ae"} Nov 01 22:47:58.451332 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.451274 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:47:58.496239 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496214 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-podinfo\") pod \"a234993f-449e-4d56-82c5-8bc2282a67fd\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " Nov 01 22:47:58.496373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496253 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-token\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-token\") pod \"a234993f-449e-4d56-82c5-8bc2282a67fd\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " Nov 01 22:47:58.496373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496286 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a234993f-449e-4d56-82c5-8bc2282a67fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a234993f-449e-4d56-82c5-8bc2282a67fd-istiod-ca-cert\") pod \"a234993f-449e-4d56-82c5-8bc2282a67fd\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " Nov 01 22:47:58.496373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496307 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-data\") pod \"a234993f-449e-4d56-82c5-8bc2282a67fd\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " Nov 01 22:47:58.496373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496351 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-credential-socket\") pod \"a234993f-449e-4d56-82c5-8bc2282a67fd\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " Nov 01 22:47:58.496591 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496372 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-envoy\") pod \"a234993f-449e-4d56-82c5-8bc2282a67fd\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " Nov 01 22:47:58.496591 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496444 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-socket\") pod \"a234993f-449e-4d56-82c5-8bc2282a67fd\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " Nov 01 22:47:58.496591 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496508 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-kube-api-access-wd7zj\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-kube-api-access-wd7zj\") pod \"a234993f-449e-4d56-82c5-8bc2282a67fd\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " Nov 01 22:47:58.496757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496609 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-certs\") pod \"a234993f-449e-4d56-82c5-8bc2282a67fd\" (UID: \"a234993f-449e-4d56-82c5-8bc2282a67fd\") " Nov 01 22:47:58.496757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496721 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-data" pod "a234993f-449e-4d56-82c5-8bc2282a67fd" (UID: "a234993f-449e-4d56-82c5-8bc2282a67fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:58.496757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496735 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-credential-socket" pod "a234993f-449e-4d56-82c5-8bc2282a67fd" (UID: "a234993f-449e-4d56-82c5-8bc2282a67fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:58.496921 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496819 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a234993f-449e-4d56-82c5-8bc2282a67fd-istiod-ca-cert" pod "a234993f-449e-4d56-82c5-8bc2282a67fd" (UID: "a234993f-449e-4d56-82c5-8bc2282a67fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:58.496996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.496974 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-socket" pod "a234993f-449e-4d56-82c5-8bc2282a67fd" (UID: "a234993f-449e-4d56-82c5-8bc2282a67fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:58.497106 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.497083 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-certs" pod "a234993f-449e-4d56-82c5-8bc2282a67fd" (UID: "a234993f-449e-4d56-82c5-8bc2282a67fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:58.497299 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.497279 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:58.497372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.497304 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a234993f-449e-4d56-82c5-8bc2282a67fd-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:58.497372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.497319 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:58.497372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.497333 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:58.497372 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.497348 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:58.498534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.498509 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-podinfo" pod "a234993f-449e-4d56-82c5-8bc2282a67fd" (UID: "a234993f-449e-4d56-82c5-8bc2282a67fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:58.498703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.498683 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-token" pod "a234993f-449e-4d56-82c5-8bc2282a67fd" (UID: "a234993f-449e-4d56-82c5-8bc2282a67fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:58.498836 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.498818 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-kube-api-access-wd7zj" pod "a234993f-449e-4d56-82c5-8bc2282a67fd" (UID: "a234993f-449e-4d56-82c5-8bc2282a67fd"). InnerVolumeSpecName "kube-api-access-wd7zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:58.498910 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.498856 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-envoy" pod "a234993f-449e-4d56-82c5-8bc2282a67fd" (UID: "a234993f-449e-4d56-82c5-8bc2282a67fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:58.598140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.598088 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:58.598140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.598130 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:58.598140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.598145 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a234993f-449e-4d56-82c5-8bc2282a67fd-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:58.598373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:58.598160 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd7zj\" (UniqueName: \"kubernetes.io/projected/a234993f-449e-4d56-82c5-8bc2282a67fd-kube-api-access-wd7zj\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:59.315041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:59.315005 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" event={"ID":"a234993f-449e-4d56-82c5-8bc2282a67fd","Type":"ContainerDied","Data":"9dcf7c068283768473cfa5a75f166f0ecece68164811bd74dbda5df7e1416020"} Nov 01 22:47:59.315501 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:59.315051 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk" Nov 01 22:47:59.315501 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:59.315093 6287 scope.go:122] "RemoveContainer" containerID="dd42c70b0feda16a153cb16450acbf5358dba85c444c04328685b180c06e07ae" Nov 01 22:47:59.343106 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:59.343082 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk"] Nov 01 22:47:59.353064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:59.353039 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-fq8qk"] Nov 01 22:47:59.404164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:59.404102 6287 patch_prober.go:29] interesting pod/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:47:59.404362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:59.404192 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:47:59.471362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:47:59.471333 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" path="/var/lib/kubelet/pods/a234993f-449e-4d56-82c5-8bc2282a67fd/volumes" Nov 01 22:48:02.342198 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.342162 6287 generic.go:356] "Generic (PLEG): container finished" podID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerID="3c686ccbc58dfd3b8af485b6c6484bf3a88152ae37de7c71d8bd0cee76703bc6" exitCode=0 Nov 01 22:48:02.342617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.342202 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" event={"ID":"b5dc5043-94c8-4546-8c7b-a9c14540f45c","Type":"ContainerDied","Data":"3c686ccbc58dfd3b8af485b6c6484bf3a88152ae37de7c71d8bd0cee76703bc6"} Nov 01 22:48:02.407899 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.407835 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:48:02.543594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.543555 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-podinfo\") pod \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " Nov 01 22:48:02.543773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.543615 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-credential-socket\") pod \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " Nov 01 22:48:02.543773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.543670 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-data\") pod \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " Nov 01 22:48:02.543773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.543720 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-token\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-token\") pod \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " Nov 01 22:48:02.543773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.543752 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-envoy\") pod \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " Nov 01 22:48:02.544016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.543805 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-kube-api-access-nc9j5\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-kube-api-access-nc9j5\") pod \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " Nov 01 22:48:02.544016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.543951 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-certs\") pod \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " Nov 01 22:48:02.544016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.543983 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-socket\") pod \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " Nov 01 22:48:02.544247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.544024 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istiod-ca-cert\") pod \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\" (UID: \"b5dc5043-94c8-4546-8c7b-a9c14540f45c\") " Nov 01 22:48:02.544247 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.544047 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-credential-socket" pod "b5dc5043-94c8-4546-8c7b-a9c14540f45c" (UID: "b5dc5043-94c8-4546-8c7b-a9c14540f45c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.544355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.544305 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-socket" pod "b5dc5043-94c8-4546-8c7b-a9c14540f45c" (UID: "b5dc5043-94c8-4546-8c7b-a9c14540f45c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.544472 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.544436 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-data" pod "b5dc5043-94c8-4546-8c7b-a9c14540f45c" (UID: "b5dc5043-94c8-4546-8c7b-a9c14540f45c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.544631 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.544609 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.544700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.544637 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.544700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.544637 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istiod-ca-cert" pod "b5dc5043-94c8-4546-8c7b-a9c14540f45c" (UID: "b5dc5043-94c8-4546-8c7b-a9c14540f45c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:02.544700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.544653 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.544880 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.544724 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-certs" pod "b5dc5043-94c8-4546-8c7b-a9c14540f45c" (UID: "b5dc5043-94c8-4546-8c7b-a9c14540f45c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.546093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.546061 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-envoy" pod "b5dc5043-94c8-4546-8c7b-a9c14540f45c" (UID: "b5dc5043-94c8-4546-8c7b-a9c14540f45c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:02.546259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.546213 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-podinfo" pod "b5dc5043-94c8-4546-8c7b-a9c14540f45c" (UID: "b5dc5043-94c8-4546-8c7b-a9c14540f45c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:48:02.546337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.546317 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-token" pod "b5dc5043-94c8-4546-8c7b-a9c14540f45c" (UID: "b5dc5043-94c8-4546-8c7b-a9c14540f45c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:02.546560 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.546541 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-kube-api-access-nc9j5" pod "b5dc5043-94c8-4546-8c7b-a9c14540f45c" (UID: "b5dc5043-94c8-4546-8c7b-a9c14540f45c"). InnerVolumeSpecName "kube-api-access-nc9j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:02.645947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.645908 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.645947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.645951 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.646189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.645966 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.646189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.645980 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.646189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.646012 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5dc5043-94c8-4546-8c7b-a9c14540f45c-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:02.646189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:02.646030 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc9j5\" (UniqueName: \"kubernetes.io/projected/b5dc5043-94c8-4546-8c7b-a9c14540f45c-kube-api-access-nc9j5\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:03.350647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:03.350614 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" event={"ID":"b5dc5043-94c8-4546-8c7b-a9c14540f45c","Type":"ContainerDied","Data":"fd17babfb90ff0cc1ec6fde080f99bc6e974f83290070f08a4f1c6a12a0f2b2d"} Nov 01 22:48:03.351143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:03.350652 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz" Nov 01 22:48:03.351143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:03.350676 6287 scope.go:122] "RemoveContainer" containerID="3c686ccbc58dfd3b8af485b6c6484bf3a88152ae37de7c71d8bd0cee76703bc6" Nov 01 22:48:03.375784 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:03.375758 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz"] Nov 01 22:48:03.381718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:03.381691 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b28a40-openshift-default-668c9c7b7-vrlsz"] Nov 01 22:48:03.470324 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:03.470288 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" path="/var/lib/kubelet/pods/b5dc5043-94c8-4546-8c7b-a9c14540f45c/volumes" Nov 01 22:48:15.587641 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:48:15.587599 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:48:17.225568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:17.225542 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 22:48:17.226187 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:17.226170 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 22:48:19.659042 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:19.659012 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 22:48:19.659666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:19.659650 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 22:48:30.929603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:30.929565 6287 scope.go:122] "RemoveContainer" containerID="94bb13335fd5335e12a612b47d2a21a4c212dc3858006ea74a47c1b5a3925cd0" Nov 01 22:48:36.024193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:36.024159 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb"] Nov 01 22:48:36.028633 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:36.028602 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2e75dd-jm5kb"] Nov 01 22:48:37.470401 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:37.470367 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad36287-197e-45f8-83f6-66780f2602c3" path="/var/lib/kubelet/pods/0ad36287-197e-45f8-83f6-66780f2602c3/volumes" Nov 01 22:48:40.780086 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780049 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x"] Nov 01 22:48:40.780541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780526 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e2dd6b25-84e7-4518-b112-3ad691d96e98" containerName="revoke-keys" Nov 01 22:48:40.780586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780542 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2dd6b25-84e7-4518-b112-3ad691d96e98" containerName="revoke-keys" Nov 01 22:48:40.780586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780555 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" Nov 01 22:48:40.780586 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780561 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" Nov 01 22:48:40.780693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780592 6287 cpu_manager.go:414] "containerMap: removing container" podUID="bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6" containerName="revoke-keys" Nov 01 22:48:40.780693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780598 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6" containerName="revoke-keys" Nov 01 22:48:40.780693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780608 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" Nov 01 22:48:40.780693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780613 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" Nov 01 22:48:40.780693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780626 6287 cpu_manager.go:414] "containerMap: removing container" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" Nov 01 22:48:40.780693 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780632 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" Nov 01 22:48:40.780891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780707 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2dd6b25-84e7-4518-b112-3ad691d96e98" containerName="revoke-keys" Nov 01 22:48:40.780891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780717 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5dc5043-94c8-4546-8c7b-a9c14540f45c" containerName="istio-proxy" Nov 01 22:48:40.780891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780730 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6" containerName="revoke-keys" Nov 01 22:48:40.780891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780739 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a234993f-449e-4d56-82c5-8bc2282a67fd" containerName="istio-proxy" Nov 01 22:48:40.780891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.780746 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b2092d-b96e-485d-922d-3322cafbcf4d" containerName="main" Nov 01 22:48:40.785199 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.785179 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:40.789394 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.789369 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:48:40.790602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.790462 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:48:40.790602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.790550 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:48:40.790602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.790576 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:48:40.792302 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.792277 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x"] Nov 01 22:48:40.865758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.865730 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw27j\" (UniqueName: \"kubernetes.io/projected/45bff268-bba1-4559-a6d8-937c1480de8a-kube-api-access-rw27j\") pod \"maas-api-revoke-sub-f9d2bd0c-2ml8x\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:40.865900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.865844 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9d2bd0c-2ml8x\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:40.865949 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.865900 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9d2bd0c-2ml8x\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:40.967476 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.967437 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw27j\" (UniqueName: \"kubernetes.io/projected/45bff268-bba1-4559-a6d8-937c1480de8a-kube-api-access-rw27j\") pod \"maas-api-revoke-sub-f9d2bd0c-2ml8x\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:40.967650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.967568 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9d2bd0c-2ml8x\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:40.967650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.967628 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9d2bd0c-2ml8x\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:40.968289 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.968268 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9d2bd0c-2ml8x\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:40.970153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.970132 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9d2bd0c-2ml8x\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:40.976573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:40.976554 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw27j\" (UniqueName: \"kubernetes.io/projected/45bff268-bba1-4559-a6d8-937c1480de8a-kube-api-access-rw27j\") pod \"maas-api-revoke-sub-f9d2bd0c-2ml8x\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:41.107143 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:41.107037 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:41.389752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:41.389724 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x"] Nov 01 22:48:41.391210 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:48:41.391181 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bff268_bba1_4559_a6d8_937c1480de8a.slice/crio-e1549da296849c538df9a1a8ff42659da778db3f137c4b4636bca6f171ca611b WatchSource:0}: Error finding container e1549da296849c538df9a1a8ff42659da778db3f137c4b4636bca6f171ca611b: Status 404 returned error can't find the container with id e1549da296849c538df9a1a8ff42659da778db3f137c4b4636bca6f171ca611b Nov 01 22:48:41.625999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:41.625963 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" event={"ID":"45bff268-bba1-4559-a6d8-937c1480de8a","Type":"ContainerStarted","Data":"e1549da296849c538df9a1a8ff42659da778db3f137c4b4636bca6f171ca611b"} Nov 01 22:48:42.634069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:42.634033 6287 generic.go:356] "Generic (PLEG): container finished" podID="45bff268-bba1-4559-a6d8-937c1480de8a" containerID="7f6bded81deb870c90cba829c0c8a826c612dbb028ac5d12c7696be10c15a527" exitCode=0 Nov 01 22:48:42.634473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:42.634124 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" event={"ID":"45bff268-bba1-4559-a6d8-937c1480de8a","Type":"ContainerDied","Data":"7f6bded81deb870c90cba829c0c8a826c612dbb028ac5d12c7696be10c15a527"} Nov 01 22:48:43.835024 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:43.834998 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:43.900160 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:43.900087 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bff268-bba1-4559-a6d8-937c1480de8a-kube-api-access-rw27j\" (UniqueName: \"kubernetes.io/projected/45bff268-bba1-4559-a6d8-937c1480de8a-kube-api-access-rw27j\") pod \"45bff268-bba1-4559-a6d8-937c1480de8a\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " Nov 01 22:48:43.900360 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:43.900304 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-openshift-ca\") pod \"45bff268-bba1-4559-a6d8-937c1480de8a\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " Nov 01 22:48:43.900438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:43.900387 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-xks-ca\") pod \"45bff268-bba1-4559-a6d8-937c1480de8a\" (UID: \"45bff268-bba1-4559-a6d8-937c1480de8a\") " Nov 01 22:48:43.900710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:43.900678 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-openshift-ca" pod "45bff268-bba1-4559-a6d8-937c1480de8a" (UID: "45bff268-bba1-4559-a6d8-937c1480de8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:43.900942 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:43.900918 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:43.902515 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:43.902490 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-xks-ca" pod "45bff268-bba1-4559-a6d8-937c1480de8a" (UID: "45bff268-bba1-4559-a6d8-937c1480de8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:43.903392 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:43.903366 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bff268-bba1-4559-a6d8-937c1480de8a-kube-api-access-rw27j" pod "45bff268-bba1-4559-a6d8-937c1480de8a" (UID: "45bff268-bba1-4559-a6d8-937c1480de8a"). InnerVolumeSpecName "kube-api-access-rw27j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:44.001761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:44.001727 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw27j\" (UniqueName: \"kubernetes.io/projected/45bff268-bba1-4559-a6d8-937c1480de8a-kube-api-access-rw27j\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:44.001761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:44.001758 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45bff268-bba1-4559-a6d8-937c1480de8a-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:44.671610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:44.671574 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" Nov 01 22:48:44.671773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:44.671607 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x" event={"ID":"45bff268-bba1-4559-a6d8-937c1480de8a","Type":"ContainerDied","Data":"e1549da296849c538df9a1a8ff42659da778db3f137c4b4636bca6f171ca611b"} Nov 01 22:48:44.671773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:44.671652 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1549da296849c538df9a1a8ff42659da778db3f137c4b4636bca6f171ca611b" Nov 01 22:48:45.588234 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:48:45.588191 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:48:52.022905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.022803 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p"] Nov 01 22:48:52.023443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.023293 6287 cpu_manager.go:414] "containerMap: removing container" podUID="45bff268-bba1-4559-a6d8-937c1480de8a" containerName="revoke-keys" Nov 01 22:48:52.023443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.023306 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bff268-bba1-4559-a6d8-937c1480de8a" containerName="revoke-keys" Nov 01 22:48:52.023443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.023385 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bff268-bba1-4559-a6d8-937c1480de8a" containerName="revoke-keys" Nov 01 22:48:52.026847 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.026827 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.036082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.036037 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:48:52.036238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.036163 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:48:52.036238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.036172 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:48:52.036369 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.036281 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:48:52.039721 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.039700 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p"] Nov 01 22:48:52.195786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.195748 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fsmf\" (UniqueName: \"kubernetes.io/projected/75b3cb34-8631-4aeb-b707-ba93f95dfb82-kube-api-access-8fsmf\") pod \"maas-api-revoke-sub-b0f8c629-c6d6p\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.195962 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.195909 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0f8c629-c6d6p\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.196126 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.196092 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0f8c629-c6d6p\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.296878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.296789 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0f8c629-c6d6p\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.297061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.296947 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0f8c629-c6d6p\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.297061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.296975 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fsmf\" (UniqueName: \"kubernetes.io/projected/75b3cb34-8631-4aeb-b707-ba93f95dfb82-kube-api-access-8fsmf\") pod \"maas-api-revoke-sub-b0f8c629-c6d6p\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.297548 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.297523 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0f8c629-c6d6p\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.299385 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.299366 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0f8c629-c6d6p\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.306835 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.306809 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fsmf\" (UniqueName: \"kubernetes.io/projected/75b3cb34-8631-4aeb-b707-ba93f95dfb82-kube-api-access-8fsmf\") pod \"maas-api-revoke-sub-b0f8c629-c6d6p\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.342750 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.342726 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:52.623410 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.623383 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p"] Nov 01 22:48:52.624640 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:48:52.624610 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b3cb34_8631_4aeb_b707_ba93f95dfb82.slice/crio-ee0de25482893f91a0b0f739e17f8f7e3136f05c8c90b2a81fc9c7a370d1ae3c WatchSource:0}: Error finding container ee0de25482893f91a0b0f739e17f8f7e3136f05c8c90b2a81fc9c7a370d1ae3c: Status 404 returned error can't find the container with id ee0de25482893f91a0b0f739e17f8f7e3136f05c8c90b2a81fc9c7a370d1ae3c Nov 01 22:48:52.739834 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:52.739803 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" event={"ID":"75b3cb34-8631-4aeb-b707-ba93f95dfb82","Type":"ContainerStarted","Data":"ee0de25482893f91a0b0f739e17f8f7e3136f05c8c90b2a81fc9c7a370d1ae3c"} Nov 01 22:48:53.748654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:53.748621 6287 generic.go:356] "Generic (PLEG): container finished" podID="75b3cb34-8631-4aeb-b707-ba93f95dfb82" containerID="5b768d446583738f5d384b1f61ea1409796373e944f1d08dc86df85afeb1de6a" exitCode=0 Nov 01 22:48:53.749132 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:53.748663 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" event={"ID":"75b3cb34-8631-4aeb-b707-ba93f95dfb82","Type":"ContainerDied","Data":"5b768d446583738f5d384b1f61ea1409796373e944f1d08dc86df85afeb1de6a"} Nov 01 22:48:54.921459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:54.921430 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:55.026211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.026131 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b3cb34-8631-4aeb-b707-ba93f95dfb82-kube-api-access-8fsmf\" (UniqueName: \"kubernetes.io/projected/75b3cb34-8631-4aeb-b707-ba93f95dfb82-kube-api-access-8fsmf\") pod \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " Nov 01 22:48:55.026211 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.026178 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-openshift-ca\") pod \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " Nov 01 22:48:55.026452 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.026235 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-xks-ca\") pod \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\" (UID: \"75b3cb34-8631-4aeb-b707-ba93f95dfb82\") " Nov 01 22:48:55.026598 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.026572 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-openshift-ca" pod "75b3cb34-8631-4aeb-b707-ba93f95dfb82" (UID: "75b3cb34-8631-4aeb-b707-ba93f95dfb82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:55.028353 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.028328 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-xks-ca" pod "75b3cb34-8631-4aeb-b707-ba93f95dfb82" (UID: "75b3cb34-8631-4aeb-b707-ba93f95dfb82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:55.028587 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.028563 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b3cb34-8631-4aeb-b707-ba93f95dfb82-kube-api-access-8fsmf" pod "75b3cb34-8631-4aeb-b707-ba93f95dfb82" (UID: "75b3cb34-8631-4aeb-b707-ba93f95dfb82"). InnerVolumeSpecName "kube-api-access-8fsmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:55.127492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.127455 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fsmf\" (UniqueName: \"kubernetes.io/projected/75b3cb34-8631-4aeb-b707-ba93f95dfb82-kube-api-access-8fsmf\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:55.127492 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.127488 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:55.127735 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.127502 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b3cb34-8631-4aeb-b707-ba93f95dfb82-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:55.764732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.764643 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" Nov 01 22:48:55.764732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.764661 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p" event={"ID":"75b3cb34-8631-4aeb-b707-ba93f95dfb82","Type":"ContainerDied","Data":"ee0de25482893f91a0b0f739e17f8f7e3136f05c8c90b2a81fc9c7a370d1ae3c"} Nov 01 22:48:55.764732 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:48:55.764719 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee0de25482893f91a0b0f739e17f8f7e3136f05c8c90b2a81fc9c7a370d1ae3c" Nov 01 22:49:02.844226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.844192 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh"] Nov 01 22:49:02.844656 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.844643 6287 cpu_manager.go:414] "containerMap: removing container" podUID="75b3cb34-8631-4aeb-b707-ba93f95dfb82" containerName="revoke-keys" Nov 01 22:49:02.844718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.844656 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b3cb34-8631-4aeb-b707-ba93f95dfb82" containerName="revoke-keys" Nov 01 22:49:02.844758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.844723 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b3cb34-8631-4aeb-b707-ba93f95dfb82" containerName="revoke-keys" Nov 01 22:49:02.848220 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.848199 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:02.856232 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.856205 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:49:02.856731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.856432 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:49:02.857277 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.857251 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:49:02.857400 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.857316 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:49:02.859278 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.859255 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh"] Nov 01 22:49:02.913061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.913026 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77ffm\" (UniqueName: \"kubernetes.io/projected/d67d0a7f-3521-4a36-b2bd-400289c2da90-kube-api-access-77ffm\") pod \"maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:02.913278 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.913083 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:02.913278 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:02.913225 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:03.014181 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.014146 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77ffm\" (UniqueName: \"kubernetes.io/projected/d67d0a7f-3521-4a36-b2bd-400289c2da90-kube-api-access-77ffm\") pod \"maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:03.014361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.014203 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:03.014361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.014257 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:03.014848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.014826 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:03.016650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.016619 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:03.030473 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.030450 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77ffm\" (UniqueName: \"kubernetes.io/projected/d67d0a7f-3521-4a36-b2bd-400289c2da90-kube-api-access-77ffm\") pod \"maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:03.168165 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.168044 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:03.447025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.446994 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh"] Nov 01 22:49:03.448044 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:49:03.448014 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67d0a7f_3521_4a36_b2bd_400289c2da90.slice/crio-6166bda7de049b6dd05e80b082c29c16db573ae2feb192d2348a4299273c037b WatchSource:0}: Error finding container 6166bda7de049b6dd05e80b082c29c16db573ae2feb192d2348a4299273c037b: Status 404 returned error can't find the container with id 6166bda7de049b6dd05e80b082c29c16db573ae2feb192d2348a4299273c037b Nov 01 22:49:03.837316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.837283 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" event={"ID":"d67d0a7f-3521-4a36-b2bd-400289c2da90","Type":"ContainerStarted","Data":"332da1cae60121fcc57f8340e51d83aa26f4c9876197ea1c02cbc5082c355916"} Nov 01 22:49:03.837454 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.837339 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" event={"ID":"d67d0a7f-3521-4a36-b2bd-400289c2da90","Type":"ContainerStarted","Data":"6166bda7de049b6dd05e80b082c29c16db573ae2feb192d2348a4299273c037b"} Nov 01 22:49:03.873212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:03.873165 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" podStartSLOduration=1.8731506169999999 podStartE2EDuration="1.873150617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:03.870771264 +0000 UTC m=+1556.519270511" watchObservedRunningTime="2026-11-01 22:49:03.873150617 +0000 UTC m=+1556.521649851" Nov 01 22:49:04.846638 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:04.846605 6287 generic.go:356] "Generic (PLEG): container finished" podID="d67d0a7f-3521-4a36-b2bd-400289c2da90" containerID="332da1cae60121fcc57f8340e51d83aa26f4c9876197ea1c02cbc5082c355916" exitCode=0 Nov 01 22:49:04.846852 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:04.846689 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" event={"ID":"d67d0a7f-3521-4a36-b2bd-400289c2da90","Type":"ContainerDied","Data":"332da1cae60121fcc57f8340e51d83aa26f4c9876197ea1c02cbc5082c355916"} Nov 01 22:49:06.036676 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.036654 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:06.149761 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.149674 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-xks-ca\") pod \"d67d0a7f-3521-4a36-b2bd-400289c2da90\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " Nov 01 22:49:06.149941 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.149757 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67d0a7f-3521-4a36-b2bd-400289c2da90-kube-api-access-77ffm\" (UniqueName: \"kubernetes.io/projected/d67d0a7f-3521-4a36-b2bd-400289c2da90-kube-api-access-77ffm\") pod \"d67d0a7f-3521-4a36-b2bd-400289c2da90\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " Nov 01 22:49:06.149941 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.149880 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-openshift-ca\") pod \"d67d0a7f-3521-4a36-b2bd-400289c2da90\" (UID: \"d67d0a7f-3521-4a36-b2bd-400289c2da90\") " Nov 01 22:49:06.150251 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.150228 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-openshift-ca" pod "d67d0a7f-3521-4a36-b2bd-400289c2da90" (UID: "d67d0a7f-3521-4a36-b2bd-400289c2da90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:06.151785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.151762 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-xks-ca" pod "d67d0a7f-3521-4a36-b2bd-400289c2da90" (UID: "d67d0a7f-3521-4a36-b2bd-400289c2da90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:06.151891 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.151872 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67d0a7f-3521-4a36-b2bd-400289c2da90-kube-api-access-77ffm" pod "d67d0a7f-3521-4a36-b2bd-400289c2da90" (UID: "d67d0a7f-3521-4a36-b2bd-400289c2da90"). InnerVolumeSpecName "kube-api-access-77ffm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:06.251596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.251562 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:06.251596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.251592 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d67d0a7f-3521-4a36-b2bd-400289c2da90-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:06.251757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.251608 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77ffm\" (UniqueName: \"kubernetes.io/projected/d67d0a7f-3521-4a36-b2bd-400289c2da90-kube-api-access-77ffm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:06.864051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.864012 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" event={"ID":"d67d0a7f-3521-4a36-b2bd-400289c2da90","Type":"ContainerDied","Data":"6166bda7de049b6dd05e80b082c29c16db573ae2feb192d2348a4299273c037b"} Nov 01 22:49:06.864051 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.864046 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh" Nov 01 22:49:06.864387 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:06.864067 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6166bda7de049b6dd05e80b082c29c16db573ae2feb192d2348a4299273c037b" Nov 01 22:49:15.588669 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:49:15.588629 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:49:15.750588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:15.744758 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh"] Nov 01 22:49:15.750588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:15.749973 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1fddbd-5bmgh"] Nov 01 22:49:17.474662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:17.474626 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67d0a7f-3521-4a36-b2bd-400289c2da90" path="/var/lib/kubelet/pods/d67d0a7f-3521-4a36-b2bd-400289c2da90/volumes" Nov 01 22:49:24.018639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.018607 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x"] Nov 01 22:49:24.019078 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.018842 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" containerID="cri-o://e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035" gracePeriod=30 Nov 01 22:49:24.565321 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.565292 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6"] Nov 01 22:49:24.565757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.565743 6287 cpu_manager.go:414] "containerMap: removing container" podUID="d67d0a7f-3521-4a36-b2bd-400289c2da90" containerName="revoke-keys" Nov 01 22:49:24.565808 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.565759 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67d0a7f-3521-4a36-b2bd-400289c2da90" containerName="revoke-keys" Nov 01 22:49:24.565856 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.565847 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67d0a7f-3521-4a36-b2bd-400289c2da90" containerName="revoke-keys" Nov 01 22:49:24.570447 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.570419 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.574977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.574955 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:49:24.575202 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.575046 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:49:24.575338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.575145 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:49:24.575338 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.575204 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:49:24.579921 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.579898 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6"] Nov 01 22:49:24.660252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.660213 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqcff\" (UniqueName: \"kubernetes.io/projected/a0701792-0b59-49d9-a130-f02369ca95f3-kube-api-access-jqcff\") pod \"maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.660639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.660618 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.660795 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.660780 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.762156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.762086 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqcff\" (UniqueName: \"kubernetes.io/projected/a0701792-0b59-49d9-a130-f02369ca95f3-kube-api-access-jqcff\") pod \"maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.762340 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.762292 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.762388 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.762354 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.763050 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.763022 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.764893 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.764863 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.771313 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.771284 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqcff\" (UniqueName: \"kubernetes.io/projected/a0701792-0b59-49d9-a130-f02369ca95f3-kube-api-access-jqcff\") pod \"maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:24.886857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:24.886787 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:25.173659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:25.173633 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6"] Nov 01 22:49:25.174065 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:49:25.173732 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0701792_0b59_49d9_a130_f02369ca95f3.slice/crio-e859ac02869dc582a45280973896949f02662b789f96261effaba0f05a0875ab WatchSource:0}: Error finding container e859ac02869dc582a45280973896949f02662b789f96261effaba0f05a0875ab: Status 404 returned error can't find the container with id e859ac02869dc582a45280973896949f02662b789f96261effaba0f05a0875ab Nov 01 22:49:26.021914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:26.021873 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" event={"ID":"a0701792-0b59-49d9-a130-f02369ca95f3","Type":"ContainerStarted","Data":"819ef50f4aeac62b9342fdbefe4d2e8719d542e590b5888d9dc41658a045f12f"} Nov 01 22:49:26.022157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:26.021933 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" event={"ID":"a0701792-0b59-49d9-a130-f02369ca95f3","Type":"ContainerStarted","Data":"e859ac02869dc582a45280973896949f02662b789f96261effaba0f05a0875ab"} Nov 01 22:49:26.040037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:26.039989 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" podStartSLOduration=2.039972269 podStartE2EDuration="2.039972269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:26.037989744 +0000 UTC m=+1578.686488980" watchObservedRunningTime="2026-11-01 22:49:26.039972269 +0000 UTC m=+1578.688471505" Nov 01 22:49:27.030972 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:27.030937 6287 generic.go:356] "Generic (PLEG): container finished" podID="a0701792-0b59-49d9-a130-f02369ca95f3" containerID="819ef50f4aeac62b9342fdbefe4d2e8719d542e590b5888d9dc41658a045f12f" exitCode=0 Nov 01 22:49:27.031405 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:27.031022 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" event={"ID":"a0701792-0b59-49d9-a130-f02369ca95f3","Type":"ContainerDied","Data":"819ef50f4aeac62b9342fdbefe4d2e8719d542e590b5888d9dc41658a045f12f"} Nov 01 22:49:28.178059 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.178031 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:28.309375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.309291 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-openshift-ca\") pod \"a0701792-0b59-49d9-a130-f02369ca95f3\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " Nov 01 22:49:28.309547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.309388 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0701792-0b59-49d9-a130-f02369ca95f3-kube-api-access-jqcff\" (UniqueName: \"kubernetes.io/projected/a0701792-0b59-49d9-a130-f02369ca95f3-kube-api-access-jqcff\") pod \"a0701792-0b59-49d9-a130-f02369ca95f3\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " Nov 01 22:49:28.309547 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.309436 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-xks-ca\") pod \"a0701792-0b59-49d9-a130-f02369ca95f3\" (UID: \"a0701792-0b59-49d9-a130-f02369ca95f3\") " Nov 01 22:49:28.309677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.309646 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-openshift-ca" pod "a0701792-0b59-49d9-a130-f02369ca95f3" (UID: "a0701792-0b59-49d9-a130-f02369ca95f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:28.311574 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.311550 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-xks-ca" pod "a0701792-0b59-49d9-a130-f02369ca95f3" (UID: "a0701792-0b59-49d9-a130-f02369ca95f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:28.311674 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.311573 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0701792-0b59-49d9-a130-f02369ca95f3-kube-api-access-jqcff" pod "a0701792-0b59-49d9-a130-f02369ca95f3" (UID: "a0701792-0b59-49d9-a130-f02369ca95f3"). InnerVolumeSpecName "kube-api-access-jqcff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:28.411330 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.411293 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqcff\" (UniqueName: \"kubernetes.io/projected/a0701792-0b59-49d9-a130-f02369ca95f3-kube-api-access-jqcff\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:28.411330 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.411325 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:28.411543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:28.411339 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0701792-0b59-49d9-a130-f02369ca95f3-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.048764 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.048734 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" Nov 01 22:49:29.048894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.048755 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6" event={"ID":"a0701792-0b59-49d9-a130-f02369ca95f3","Type":"ContainerDied","Data":"e859ac02869dc582a45280973896949f02662b789f96261effaba0f05a0875ab"} Nov 01 22:49:29.048894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.048812 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e859ac02869dc582a45280973896949f02662b789f96261effaba0f05a0875ab" Nov 01 22:49:29.282046 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.282021 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:49:29.423256 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423178 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-kube-api-access-2h44d\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-kube-api-access-2h44d\") pod \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " Nov 01 22:49:29.423256 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423227 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-envoy\") pod \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " Nov 01 22:49:29.423455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423265 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-token\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-token\") pod \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " Nov 01 22:49:29.423455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423292 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-socket\") pod \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " Nov 01 22:49:29.423455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423309 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-data\") pod \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " Nov 01 22:49:29.423455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423367 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istiod-ca-cert\") pod \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " Nov 01 22:49:29.423455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423435 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-credential-socket\") pod \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " Nov 01 22:49:29.423743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423479 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-podinfo\") pod \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " Nov 01 22:49:29.423743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423581 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-socket" pod "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" (UID: "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.423743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423666 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-certs\") pod \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\" (UID: \"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2\") " Nov 01 22:49:29.423743 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423671 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-data" pod "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" (UID: "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.423965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423749 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-credential-socket" pod "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" (UID: "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.423965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.423812 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istiod-ca-cert" pod "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" (UID: "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:29.424088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.424007 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-certs" pod "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" (UID: "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.424217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.424195 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.424290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.424222 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.424290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.424236 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.424290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.424250 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.424290 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.424263 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.425567 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.425545 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-token" pod "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" (UID: "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:29.425692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.425675 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-podinfo" pod "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" (UID: "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:29.425914 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.425887 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-envoy" pod "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" (UID: "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:29.425996 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.425900 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-kube-api-access-2h44d" pod "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" (UID: "b7f44cf5-7696-47e4-a64b-4a2c7dac51c2"). InnerVolumeSpecName "kube-api-access-2h44d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:29.525459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.525429 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.525459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.525459 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h44d\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-kube-api-access-2h44d\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.525459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.525468 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:29.525702 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:29.525477 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:30.057621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.057578 6287 generic.go:356] "Generic (PLEG): container finished" podID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerID="e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035" exitCode=0 Nov 01 22:49:30.057621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.057625 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" event={"ID":"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2","Type":"ContainerDied","Data":"e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035"} Nov 01 22:49:30.057865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.057639 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" Nov 01 22:49:30.057865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.057675 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x" event={"ID":"b7f44cf5-7696-47e4-a64b-4a2c7dac51c2","Type":"ContainerDied","Data":"3c347d12ffeb4fbf8438176f65631a0dc78bdadc3a53123ef1554f1954890f43"} Nov 01 22:49:30.057865 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.057700 6287 scope.go:122] "RemoveContainer" containerID="e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035" Nov 01 22:49:30.069803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.069781 6287 scope.go:122] "RemoveContainer" containerID="e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035" Nov 01 22:49:30.070087 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:49:30.070068 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035\": container with ID starting with e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035 not found: ID does not exist" containerID="e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035" Nov 01 22:49:30.070157 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.070100 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035"} err="failed to get container status \"e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035\": rpc error: code = NotFound desc = could not find container \"e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035\": container with ID starting with e44f97feec2982d31c30094bbca54b7b262a924524d2f9877f8671186ec96035 not found: ID does not exist" Nov 01 22:49:30.078994 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.078958 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x"] Nov 01 22:49:30.082519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.082497 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1fddbd-openshift-default-689959cbb5-wlx8x"] Nov 01 22:49:30.972104 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:30.972071 6287 scope.go:122] "RemoveContainer" containerID="6c0c30bcd9cc14f5342ca8dab5e005c2bccd378b86e34ea21abeb8c5920be7f4" Nov 01 22:49:31.469848 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:31.469817 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" path="/var/lib/kubelet/pods/b7f44cf5-7696-47e4-a64b-4a2c7dac51c2/volumes" Nov 01 22:49:36.583731 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:36.583693 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6"] Nov 01 22:49:36.594645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:36.594613 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9717e2-p4bg6"] Nov 01 22:49:37.471109 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:37.471074 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0701792-0b59-49d9-a130-f02369ca95f3" path="/var/lib/kubelet/pods/a0701792-0b59-49d9-a130-f02369ca95f3/volumes" Nov 01 22:49:43.145601 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:43.145568 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l"] Nov 01 22:49:43.153189 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:43.153160 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp"] Nov 01 22:49:43.156337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:43.156310 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402f0e44-sp78l"] Nov 01 22:49:43.163493 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:43.163464 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd98065-xcmvp"] Nov 01 22:49:43.470444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:43.470370 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81db0289-d89d-4920-b91e-a6a73723d01e" path="/var/lib/kubelet/pods/81db0289-d89d-4920-b91e-a6a73723d01e/volumes" Nov 01 22:49:43.470808 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:43.470793 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57a9e61-28ff-423b-815b-52bbca44a23f" path="/var/lib/kubelet/pods/f57a9e61-28ff-423b-815b-52bbca44a23f/volumes" Nov 01 22:49:45.587752 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:49:45.587713 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:49:46.615662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.615626 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc"] Nov 01 22:49:46.616080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.616067 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a0701792-0b59-49d9-a130-f02369ca95f3" containerName="revoke-keys" Nov 01 22:49:46.616140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.616081 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0701792-0b59-49d9-a130-f02369ca95f3" containerName="revoke-keys" Nov 01 22:49:46.616140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.616097 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" Nov 01 22:49:46.616140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.616102 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" Nov 01 22:49:46.616242 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.616190 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0701792-0b59-49d9-a130-f02369ca95f3" containerName="revoke-keys" Nov 01 22:49:46.616242 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.616200 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f44cf5-7696-47e4-a64b-4a2c7dac51c2" containerName="istio-proxy" Nov 01 22:49:46.621159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.621143 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.627275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.626103 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:49:46.627275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.626610 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:49:46.627275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.626787 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:49:46.627275 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.626822 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:49:46.629516 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.629488 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc"] Nov 01 22:49:46.706320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.706286 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlv8\" (UniqueName: \"kubernetes.io/projected/a2e28904-c050-4747-8a68-fac0d62134e8-kube-api-access-7hlv8\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.706508 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.706434 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.706561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.706511 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.807435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.807402 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.807611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.807465 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlv8\" (UniqueName: \"kubernetes.io/projected/a2e28904-c050-4747-8a68-fac0d62134e8-kube-api-access-7hlv8\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.807611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.807577 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.808029 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.808006 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.810015 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.809986 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.817505 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.817481 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlv8\" (UniqueName: \"kubernetes.io/projected/a2e28904-c050-4747-8a68-fac0d62134e8-kube-api-access-7hlv8\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:46.938839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:46.938758 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:47.227152 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:49:47.227101 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e28904_c050_4747_8a68_fac0d62134e8.slice/crio-c26fb6c6c7ec2124121edbb7bcd73e427787c2b637c06005b87c7a38370f262d WatchSource:0}: Error finding container c26fb6c6c7ec2124121edbb7bcd73e427787c2b637c06005b87c7a38370f262d: Status 404 returned error can't find the container with id c26fb6c6c7ec2124121edbb7bcd73e427787c2b637c06005b87c7a38370f262d Nov 01 22:49:47.232723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:47.232696 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc"] Nov 01 22:49:48.198643 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:48.198602 6287 generic.go:356] "Generic (PLEG): container finished" podID="a2e28904-c050-4747-8a68-fac0d62134e8" containerID="758001c1fd685931293a821e68f01f17475e37284c633b8da13f4cae6ecc1d2b" exitCode=0 Nov 01 22:49:48.199105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:48.198662 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" event={"ID":"a2e28904-c050-4747-8a68-fac0d62134e8","Type":"ContainerDied","Data":"758001c1fd685931293a821e68f01f17475e37284c633b8da13f4cae6ecc1d2b"} Nov 01 22:49:48.199105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:48.198705 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" event={"ID":"a2e28904-c050-4747-8a68-fac0d62134e8","Type":"ContainerStarted","Data":"c26fb6c6c7ec2124121edbb7bcd73e427787c2b637c06005b87c7a38370f262d"} Nov 01 22:49:49.344455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.344432 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:49.434420 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.434390 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-openshift-ca\") pod \"a2e28904-c050-4747-8a68-fac0d62134e8\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " Nov 01 22:49:49.434617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.434515 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-xks-ca\") pod \"a2e28904-c050-4747-8a68-fac0d62134e8\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " Nov 01 22:49:49.434617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.434538 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e28904-c050-4747-8a68-fac0d62134e8-kube-api-access-7hlv8\" (UniqueName: \"kubernetes.io/projected/a2e28904-c050-4747-8a68-fac0d62134e8-kube-api-access-7hlv8\") pod \"a2e28904-c050-4747-8a68-fac0d62134e8\" (UID: \"a2e28904-c050-4747-8a68-fac0d62134e8\") " Nov 01 22:49:49.434824 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.434798 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-openshift-ca" pod "a2e28904-c050-4747-8a68-fac0d62134e8" (UID: "a2e28904-c050-4747-8a68-fac0d62134e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:49.436634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.436610 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e28904-c050-4747-8a68-fac0d62134e8-kube-api-access-7hlv8" pod "a2e28904-c050-4747-8a68-fac0d62134e8" (UID: "a2e28904-c050-4747-8a68-fac0d62134e8"). InnerVolumeSpecName "kube-api-access-7hlv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:49.436737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.436637 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-xks-ca" pod "a2e28904-c050-4747-8a68-fac0d62134e8" (UID: "a2e28904-c050-4747-8a68-fac0d62134e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:49.536282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.536254 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:49.536282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.536283 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlv8\" (UniqueName: \"kubernetes.io/projected/a2e28904-c050-4747-8a68-fac0d62134e8-kube-api-access-7hlv8\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:49.536437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:49.536298 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e28904-c050-4747-8a68-fac0d62134e8-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:50.214832 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:50.214802 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" Nov 01 22:49:50.215039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:50.214847 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc" event={"ID":"a2e28904-c050-4747-8a68-fac0d62134e8","Type":"ContainerDied","Data":"c26fb6c6c7ec2124121edbb7bcd73e427787c2b637c06005b87c7a38370f262d"} Nov 01 22:49:50.215039 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:50.214908 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26fb6c6c7ec2124121edbb7bcd73e427787c2b637c06005b87c7a38370f262d" Nov 01 22:49:56.730943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:56.730909 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj"] Nov 01 22:49:59.161018 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:59.160986 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc"] Nov 01 22:49:59.166450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:59.166422 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-93d36a-qhkfc"] Nov 01 22:49:59.470022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:49:59.469947 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e28904-c050-4747-8a68-fac0d62134e8" path="/var/lib/kubelet/pods/a2e28904-c050-4747-8a68-fac0d62134e8/volumes" Nov 01 22:50:01.731843 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:01.731783 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" podUID="558e040b-84a9-47f0-a2b9-edcde5995e68" containerName="payload-processing" containerID="cri-o://6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269" gracePeriod=25 Nov 01 22:50:01.999920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:01.999887 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:50:02.185543 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.185509 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558e040b-84a9-47f0-a2b9-edcde5995e68-kube-api-access-wlqf6\" (UniqueName: \"kubernetes.io/projected/558e040b-84a9-47f0-a2b9-edcde5995e68-kube-api-access-wlqf6\") pod \"558e040b-84a9-47f0-a2b9-edcde5995e68\" (UID: \"558e040b-84a9-47f0-a2b9-edcde5995e68\") " Nov 01 22:50:02.185720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.185556 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558e040b-84a9-47f0-a2b9-edcde5995e68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558e040b-84a9-47f0-a2b9-edcde5995e68-plugins-config-volume\") pod \"558e040b-84a9-47f0-a2b9-edcde5995e68\" (UID: \"558e040b-84a9-47f0-a2b9-edcde5995e68\") " Nov 01 22:50:02.185968 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.185938 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558e040b-84a9-47f0-a2b9-edcde5995e68-plugins-config-volume" pod "558e040b-84a9-47f0-a2b9-edcde5995e68" (UID: "558e040b-84a9-47f0-a2b9-edcde5995e68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:02.187771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.187748 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558e040b-84a9-47f0-a2b9-edcde5995e68-kube-api-access-wlqf6" pod "558e040b-84a9-47f0-a2b9-edcde5995e68" (UID: "558e040b-84a9-47f0-a2b9-edcde5995e68"). InnerVolumeSpecName "kube-api-access-wlqf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:02.287140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.287087 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlqf6\" (UniqueName: \"kubernetes.io/projected/558e040b-84a9-47f0-a2b9-edcde5995e68-kube-api-access-wlqf6\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:02.287140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.287137 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558e040b-84a9-47f0-a2b9-edcde5995e68-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:02.298696 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.298665 6287 generic.go:356] "Generic (PLEG): container finished" podID="558e040b-84a9-47f0-a2b9-edcde5995e68" containerID="6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269" exitCode=0 Nov 01 22:50:02.298825 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.298721 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" Nov 01 22:50:02.298825 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.298748 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" event={"ID":"558e040b-84a9-47f0-a2b9-edcde5995e68","Type":"ContainerDied","Data":"6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269"} Nov 01 22:50:02.298825 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.298812 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj" event={"ID":"558e040b-84a9-47f0-a2b9-edcde5995e68","Type":"ContainerDied","Data":"f7e55118feb881195c682faa6334fdd9e273d9135cb04ba96c08cc5d9d676c93"} Nov 01 22:50:02.298960 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.298841 6287 scope.go:122] "RemoveContainer" containerID="6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269" Nov 01 22:50:02.310164 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.310147 6287 scope.go:122] "RemoveContainer" containerID="6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269" Nov 01 22:50:02.310466 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:50:02.310441 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269\": container with ID starting with 6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269 not found: ID does not exist" containerID="6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269" Nov 01 22:50:02.310518 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.310477 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269"} err="failed to get container status \"6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269\": rpc error: code = NotFound desc = could not find container \"6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269\": container with ID starting with 6602f639cb4883dc820d7a5cb27dc23e5d5a99f6d22d95dd5a23dd85be79a269 not found: ID does not exist" Nov 01 22:50:02.322926 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.322898 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj"] Nov 01 22:50:02.326033 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:02.326009 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-93d36a-7c7f8dbb8c-tggfj"] Nov 01 22:50:03.469491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:03.469457 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558e040b-84a9-47f0-a2b9-edcde5995e68" path="/var/lib/kubelet/pods/558e040b-84a9-47f0-a2b9-edcde5995e68/volumes" Nov 01 22:50:07.249678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.249644 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6"] Nov 01 22:50:07.250193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.250102 6287 cpu_manager.go:414] "containerMap: removing container" podUID="558e040b-84a9-47f0-a2b9-edcde5995e68" containerName="payload-processing" Nov 01 22:50:07.250193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.250130 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558e040b-84a9-47f0-a2b9-edcde5995e68" containerName="payload-processing" Nov 01 22:50:07.250193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.250151 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a2e28904-c050-4747-8a68-fac0d62134e8" containerName="revoke-keys" Nov 01 22:50:07.250193 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.250157 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e28904-c050-4747-8a68-fac0d62134e8" containerName="revoke-keys" Nov 01 22:50:07.250436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.250226 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="558e040b-84a9-47f0-a2b9-edcde5995e68" containerName="payload-processing" Nov 01 22:50:07.250436 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.250235 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e28904-c050-4747-8a68-fac0d62134e8" containerName="revoke-keys" Nov 01 22:50:07.255159 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.255135 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.261416 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.261393 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:50:07.262766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.262737 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:50:07.262909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.262833 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6"] Nov 01 22:50:07.263020 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.263004 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:50:07.263083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.263062 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:50:07.342680 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.342639 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.342862 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.342755 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.342862 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.342823 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrsm\" (UniqueName: \"kubernetes.io/projected/adb768ec-89c5-43f0-9c18-8d96741e91f1-kube-api-access-lrrsm\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.444064 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.444027 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.444276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.444089 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.444276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.444257 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrsm\" (UniqueName: \"kubernetes.io/projected/adb768ec-89c5-43f0-9c18-8d96741e91f1-kube-api-access-lrrsm\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.444755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.444733 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.446504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.446485 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.453276 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.453258 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrsm\" (UniqueName: \"kubernetes.io/projected/adb768ec-89c5-43f0-9c18-8d96741e91f1-kube-api-access-lrrsm\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.571561 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.571481 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:07.848975 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:07.848943 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6"] Nov 01 22:50:07.850531 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:50:07.850501 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb768ec_89c5_43f0_9c18_8d96741e91f1.slice/crio-e188bbe35e08f7943d2d0fb637f845aabbd3694d0a21f148a1cc8eaa196c13a2 WatchSource:0}: Error finding container e188bbe35e08f7943d2d0fb637f845aabbd3694d0a21f148a1cc8eaa196c13a2: Status 404 returned error can't find the container with id e188bbe35e08f7943d2d0fb637f845aabbd3694d0a21f148a1cc8eaa196c13a2 Nov 01 22:50:08.345528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:08.345493 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" event={"ID":"adb768ec-89c5-43f0-9c18-8d96741e91f1","Type":"ContainerStarted","Data":"d7035918b6cd878d51ba5b04c8c4512c5c4cdad19edd0b1b5a3aee52657dbe68"} Nov 01 22:50:08.345956 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:08.345547 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" event={"ID":"adb768ec-89c5-43f0-9c18-8d96741e91f1","Type":"ContainerStarted","Data":"e188bbe35e08f7943d2d0fb637f845aabbd3694d0a21f148a1cc8eaa196c13a2"} Nov 01 22:50:08.361453 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:08.361412 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" podStartSLOduration=1.361397147 podStartE2EDuration="1.361397147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:08.36116553 +0000 UTC m=+1621.009664764" watchObservedRunningTime="2026-11-01 22:50:08.361397147 +0000 UTC m=+1621.009896379" Nov 01 22:50:09.354800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:09.354769 6287 generic.go:356] "Generic (PLEG): container finished" podID="adb768ec-89c5-43f0-9c18-8d96741e91f1" containerID="d7035918b6cd878d51ba5b04c8c4512c5c4cdad19edd0b1b5a3aee52657dbe68" exitCode=0 Nov 01 22:50:09.355217 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:09.354857 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" event={"ID":"adb768ec-89c5-43f0-9c18-8d96741e91f1","Type":"ContainerDied","Data":"d7035918b6cd878d51ba5b04c8c4512c5c4cdad19edd0b1b5a3aee52657dbe68"} Nov 01 22:50:10.502823 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.502797 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:10.574056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.574031 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-openshift-ca\") pod \"adb768ec-89c5-43f0-9c18-8d96741e91f1\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " Nov 01 22:50:10.574225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.574071 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-xks-ca\") pod \"adb768ec-89c5-43f0-9c18-8d96741e91f1\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " Nov 01 22:50:10.574225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.574097 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb768ec-89c5-43f0-9c18-8d96741e91f1-kube-api-access-lrrsm\" (UniqueName: \"kubernetes.io/projected/adb768ec-89c5-43f0-9c18-8d96741e91f1-kube-api-access-lrrsm\") pod \"adb768ec-89c5-43f0-9c18-8d96741e91f1\" (UID: \"adb768ec-89c5-43f0-9c18-8d96741e91f1\") " Nov 01 22:50:10.574461 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.574433 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-openshift-ca" pod "adb768ec-89c5-43f0-9c18-8d96741e91f1" (UID: "adb768ec-89c5-43f0-9c18-8d96741e91f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:10.576238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.576213 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb768ec-89c5-43f0-9c18-8d96741e91f1-kube-api-access-lrrsm" pod "adb768ec-89c5-43f0-9c18-8d96741e91f1" (UID: "adb768ec-89c5-43f0-9c18-8d96741e91f1"). InnerVolumeSpecName "kube-api-access-lrrsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:10.576353 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.576286 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-xks-ca" pod "adb768ec-89c5-43f0-9c18-8d96741e91f1" (UID: "adb768ec-89c5-43f0-9c18-8d96741e91f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:10.675406 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.675342 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:10.675406 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.675368 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb768ec-89c5-43f0-9c18-8d96741e91f1-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:10.675406 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:10.675378 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrsm\" (UniqueName: \"kubernetes.io/projected/adb768ec-89c5-43f0-9c18-8d96741e91f1-kube-api-access-lrrsm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:11.370534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:11.370504 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" Nov 01 22:50:11.370726 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:11.370544 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6" event={"ID":"adb768ec-89c5-43f0-9c18-8d96741e91f1","Type":"ContainerDied","Data":"e188bbe35e08f7943d2d0fb637f845aabbd3694d0a21f148a1cc8eaa196c13a2"} Nov 01 22:50:11.370726 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:11.370596 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e188bbe35e08f7943d2d0fb637f845aabbd3694d0a21f148a1cc8eaa196c13a2" Nov 01 22:50:15.589089 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:50:15.589055 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:50:18.395791 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:18.395691 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6"] Nov 01 22:50:18.416575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:18.416540 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-dcd5d4-2bvh6"] Nov 01 22:50:19.470129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:19.470087 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb768ec-89c5-43f0-9c18-8d96741e91f1" path="/var/lib/kubelet/pods/adb768ec-89c5-43f0-9c18-8d96741e91f1/volumes" Nov 01 22:50:25.839129 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:25.839073 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq"] Nov 01 22:50:25.841817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:25.839308 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" containerID="cri-o://15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58" gracePeriod=30 Nov 01 22:50:25.843753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:25.843725 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht"] Nov 01 22:50:25.844003 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:25.843954 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" containerID="cri-o://66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667" gracePeriod=30 Nov 01 22:50:29.115544 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:29.115503 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:50:29.115944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:29.115593 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:50:29.631772 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:29.631732 6287 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:50:29.631958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:29.631798 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:50:30.036839 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:30.036791 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr"] Nov 01 22:50:30.039801 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:30.039772 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b72845ad-xp9hr"] Nov 01 22:50:31.007616 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.007479 6287 scope.go:122] "RemoveContainer" containerID="66c2c7c85b7d7c579f083896617e4cfb57dcbb29d7520a6caedea208fb6c0be2" Nov 01 22:50:31.019575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.019534 6287 scope.go:122] "RemoveContainer" containerID="656dd563d4d7330eaf04d10bbe9cb674c1100e73f3ab06cedf688ab0a3962781" Nov 01 22:50:31.033594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.033514 6287 scope.go:122] "RemoveContainer" containerID="81549149daa3f24be3a5bce597dd54f439f62555d510d8a18332d4559ee065a8" Nov 01 22:50:31.047330 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.047305 6287 scope.go:122] "RemoveContainer" containerID="6e1da989cadace5964763a4a3e632cfee48d1b36168faa281eceadb8eb66640e" Nov 01 22:50:31.072133 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.072080 6287 scope.go:122] "RemoveContainer" containerID="84f9d005f085a3286414dbf62d982cc72505617fd8882c522b5a9bd38594f431" Nov 01 22:50:31.098333 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.096412 6287 scope.go:122] "RemoveContainer" containerID="f9c509da9a833d89d77807d307cb416dee4072583a498238bb025c08028e4676" Nov 01 22:50:31.137537 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.137514 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:50:31.157262 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.157233 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:50:31.193769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.193739 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-certs\") pod \"c8b38708-4a75-40df-9835-d15d80301461\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " Nov 01 22:50:31.193885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.193792 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-socket\") pod \"c8b38708-4a75-40df-9835-d15d80301461\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " Nov 01 22:50:31.193885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.193876 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-credential-socket\") pod \"c8b38708-4a75-40df-9835-d15d80301461\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " Nov 01 22:50:31.194063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.193911 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-kube-api-access-2tcbb\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-kube-api-access-2tcbb\") pod \"c8b38708-4a75-40df-9835-d15d80301461\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " Nov 01 22:50:31.194063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.193951 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-data\") pod \"c8b38708-4a75-40df-9835-d15d80301461\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " Nov 01 22:50:31.194063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.193981 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-istio-token\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-istio-token\") pod \"c8b38708-4a75-40df-9835-d15d80301461\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " Nov 01 22:50:31.194266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194092 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b38708-4a75-40df-9835-d15d80301461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b38708-4a75-40df-9835-d15d80301461-istiod-ca-cert\") pod \"c8b38708-4a75-40df-9835-d15d80301461\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " Nov 01 22:50:31.194266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194105 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-socket" pod "c8b38708-4a75-40df-9835-d15d80301461" (UID: "c8b38708-4a75-40df-9835-d15d80301461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.194266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194149 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-certs" pod "c8b38708-4a75-40df-9835-d15d80301461" (UID: "c8b38708-4a75-40df-9835-d15d80301461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.194266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194172 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-envoy\") pod \"c8b38708-4a75-40df-9835-d15d80301461\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " Nov 01 22:50:31.194266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194212 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8b38708-4a75-40df-9835-d15d80301461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b38708-4a75-40df-9835-d15d80301461-istio-podinfo\") pod \"c8b38708-4a75-40df-9835-d15d80301461\" (UID: \"c8b38708-4a75-40df-9835-d15d80301461\") " Nov 01 22:50:31.194266 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194223 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-credential-socket" pod "c8b38708-4a75-40df-9835-d15d80301461" (UID: "c8b38708-4a75-40df-9835-d15d80301461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.194619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194347 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-data" pod "c8b38708-4a75-40df-9835-d15d80301461" (UID: "c8b38708-4a75-40df-9835-d15d80301461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.194619 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194534 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b38708-4a75-40df-9835-d15d80301461-istiod-ca-cert" pod "c8b38708-4a75-40df-9835-d15d80301461" (UID: "c8b38708-4a75-40df-9835-d15d80301461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:31.194725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194639 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b38708-4a75-40df-9835-d15d80301461-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.194725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194656 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.194725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194676 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.194725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194695 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.194725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.194707 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.196487 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.196461 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-envoy" pod "c8b38708-4a75-40df-9835-d15d80301461" (UID: "c8b38708-4a75-40df-9835-d15d80301461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.196588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.196522 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8b38708-4a75-40df-9835-d15d80301461-istio-podinfo" pod "c8b38708-4a75-40df-9835-d15d80301461" (UID: "c8b38708-4a75-40df-9835-d15d80301461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:50:31.196729 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.196716 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-istio-token" pod "c8b38708-4a75-40df-9835-d15d80301461" (UID: "c8b38708-4a75-40df-9835-d15d80301461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:31.196958 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.196936 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-kube-api-access-2tcbb" pod "c8b38708-4a75-40df-9835-d15d80301461" (UID: "c8b38708-4a75-40df-9835-d15d80301461"). InnerVolumeSpecName "kube-api-access-2tcbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:31.287550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.287519 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5"] Nov 01 22:50:31.295325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295297 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istiod-ca-cert\") pod \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " Nov 01 22:50:31.295483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295427 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-certs\") pod \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " Nov 01 22:50:31.295483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295460 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-credential-socket\") pod \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " Nov 01 22:50:31.295647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295502 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-data\") pod \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " Nov 01 22:50:31.295647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295541 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-socket\") pod \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " Nov 01 22:50:31.295647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295579 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-kube-api-access-z8kjq\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-kube-api-access-z8kjq\") pod \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " Nov 01 22:50:31.295647 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295609 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-envoy\") pod \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " Nov 01 22:50:31.295883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295653 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-token\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-token\") pod \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " Nov 01 22:50:31.295883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295698 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istiod-ca-cert" pod "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" (UID: "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:31.295883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295718 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-podinfo\") pod \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\" (UID: \"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186\") " Nov 01 22:50:31.295883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.295840 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-credential-socket" pod "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" (UID: "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.296337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.296150 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b38708-4a75-40df-9835-d15d80301461-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.296337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.296179 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.296337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.296195 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b38708-4a75-40df-9835-d15d80301461-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.296337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.296208 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.296337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.296224 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tcbb\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-kube-api-access-2tcbb\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.296337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.296242 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b38708-4a75-40df-9835-d15d80301461-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.296337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.296280 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-data" pod "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" (UID: "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.296337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.296296 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-certs" pod "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" (UID: "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.296337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.296311 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-socket" pod "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" (UID: "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.298002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.297972 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-envoy" pod "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" (UID: "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:31.298176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.298075 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-kube-api-access-z8kjq" pod "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" (UID: "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186"). InnerVolumeSpecName "kube-api-access-z8kjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:31.298306 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.298287 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-token" pod "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" (UID: "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:31.298419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.298400 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-podinfo" pod "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" (UID: "2f18200f-e7cd-4aa5-a2b8-3d81b84b0186"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:50:31.306810 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.306787 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx"] Nov 01 22:50:31.314243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.314221 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs"] Nov 01 22:50:31.397588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.397504 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.397588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.397534 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.397588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.397544 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.397588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.397556 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8kjq\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-kube-api-access-z8kjq\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.397588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.397565 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.397588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.397573 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.397588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.397581 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:31.476037 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.476000 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e9298d-5d38-4620-815b-48910f1d13fd" path="/var/lib/kubelet/pods/58e9298d-5d38-4620-815b-48910f1d13fd/volumes" Nov 01 22:50:31.521386 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.521357 6287 generic.go:356] "Generic (PLEG): container finished" podID="c8b38708-4a75-40df-9835-d15d80301461" containerID="15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58" exitCode=0 Nov 01 22:50:31.521528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.521415 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" Nov 01 22:50:31.521528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.521439 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" event={"ID":"c8b38708-4a75-40df-9835-d15d80301461","Type":"ContainerDied","Data":"15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58"} Nov 01 22:50:31.521528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.521491 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq" event={"ID":"c8b38708-4a75-40df-9835-d15d80301461","Type":"ContainerDied","Data":"213c3061742161b125806e59d7528ad96c8c190e7474a3c059816b77255560a7"} Nov 01 22:50:31.521528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.521521 6287 scope.go:122] "RemoveContainer" containerID="15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58" Nov 01 22:50:31.523087 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.523063 6287 generic.go:356] "Generic (PLEG): container finished" podID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerID="66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667" exitCode=0 Nov 01 22:50:31.523228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.523126 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" Nov 01 22:50:31.523228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.523160 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" event={"ID":"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186","Type":"ContainerDied","Data":"66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667"} Nov 01 22:50:31.523228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.523213 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht" event={"ID":"2f18200f-e7cd-4aa5-a2b8-3d81b84b0186","Type":"ContainerDied","Data":"bfbc0b9fd77f0e8828288bd17ecf1796e571203b4e750c03deb143c3a8f9e560"} Nov 01 22:50:31.533052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.533035 6287 scope.go:122] "RemoveContainer" containerID="15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58" Nov 01 22:50:31.533334 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:50:31.533307 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58\": container with ID starting with 15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58 not found: ID does not exist" containerID="15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58" Nov 01 22:50:31.533384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.533342 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58"} err="failed to get container status \"15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58\": rpc error: code = NotFound desc = could not find container \"15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58\": container with ID starting with 15ed570d4239c3c303a54b0e0638233483610467b24a15f7d641f323ede6ca58 not found: ID does not exist" Nov 01 22:50:31.533384 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.533358 6287 scope.go:122] "RemoveContainer" containerID="66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667" Nov 01 22:50:31.542441 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.542416 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq"] Nov 01 22:50:31.546282 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.546262 6287 scope.go:122] "RemoveContainer" containerID="66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667" Nov 01 22:50:31.546610 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:50:31.546586 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667\": container with ID starting with 66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667 not found: ID does not exist" containerID="66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667" Nov 01 22:50:31.546688 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.546615 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667"} err="failed to get container status \"66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667\": rpc error: code = NotFound desc = could not find container \"66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667\": container with ID starting with 66f7d07886d59b3adeb479426b5c17ecbe8286c3b02108dd87383f3d6011c667 not found: ID does not exist" Nov 01 22:50:31.547177 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.547159 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-8klbq"] Nov 01 22:50:31.563365 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.563335 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht"] Nov 01 22:50:31.566008 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:31.565986 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dcd5d4-openshift-default-878f56b9f-7rfht"] Nov 01 22:50:32.086363 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:32.086327 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc"] Nov 01 22:50:32.091885 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:32.091859 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6"] Nov 01 22:50:32.100319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:32.100294 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c6bb954-dh7c6"] Nov 01 22:50:32.114594 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:32.114573 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ec6a8a-72phc"] Nov 01 22:50:33.470448 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:33.470418 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" path="/var/lib/kubelet/pods/2f18200f-e7cd-4aa5-a2b8-3d81b84b0186/volumes" Nov 01 22:50:33.470940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:33.470859 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b726ae8-f84c-4208-b08c-cb8c34380870" path="/var/lib/kubelet/pods/5b726ae8-f84c-4208-b08c-cb8c34380870/volumes" Nov 01 22:50:33.471304 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:33.471290 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b38708-4a75-40df-9835-d15d80301461" path="/var/lib/kubelet/pods/c8b38708-4a75-40df-9835-d15d80301461/volumes" Nov 01 22:50:33.471955 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:33.471941 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3140a23-6ae1-4c8a-b7ee-512f12880378" path="/var/lib/kubelet/pods/f3140a23-6ae1-4c8a-b7ee-512f12880378/volumes" Nov 01 22:50:35.009934 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.009896 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j"] Nov 01 22:50:35.010395 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.010149 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" podUID="2957d06d-fec2-40f0-a4b4-b433ad6ac848" containerName="discovery" containerID="cri-o://ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681" gracePeriod=30 Nov 01 22:50:35.260758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.260693 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:50:35.334317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.334284 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2957d06d-fec2-40f0-a4b4-b433ad6ac848-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2957d06d-fec2-40f0-a4b4-b433ad6ac848-local-certs\") pod \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " Nov 01 22:50:35.334317 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.334332 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-kube-api-access-22fbs\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-kube-api-access-22fbs\") pod \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " Nov 01 22:50:35.334649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.334357 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-dns-cert\") pod \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " Nov 01 22:50:35.334649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.334416 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-token\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-token\") pod \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " Nov 01 22:50:35.334649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.334502 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-kubeconfig\") pod \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " Nov 01 22:50:35.334649 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.334642 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-cacerts\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-cacerts\") pod \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " Nov 01 22:50:35.334883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.334709 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-ca-configmap\") pod \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\" (UID: \"2957d06d-fec2-40f0-a4b4-b433ad6ac848\") " Nov 01 22:50:35.335450 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.335424 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-ca-configmap" pod "2957d06d-fec2-40f0-a4b4-b433ad6ac848" (UID: "2957d06d-fec2-40f0-a4b4-b433ad6ac848"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:35.337762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.337716 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-kubeconfig" pod "2957d06d-fec2-40f0-a4b4-b433ad6ac848" (UID: "2957d06d-fec2-40f0-a4b4-b433ad6ac848"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:35.337762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.337731 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-token" pod "2957d06d-fec2-40f0-a4b4-b433ad6ac848" (UID: "2957d06d-fec2-40f0-a4b4-b433ad6ac848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:35.337985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.337894 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-cacerts" pod "2957d06d-fec2-40f0-a4b4-b433ad6ac848" (UID: "2957d06d-fec2-40f0-a4b4-b433ad6ac848"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:35.337985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.337926 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-dns-cert" pod "2957d06d-fec2-40f0-a4b4-b433ad6ac848" (UID: "2957d06d-fec2-40f0-a4b4-b433ad6ac848"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:35.337985 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.337958 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2957d06d-fec2-40f0-a4b4-b433ad6ac848-local-certs" pod "2957d06d-fec2-40f0-a4b4-b433ad6ac848" (UID: "2957d06d-fec2-40f0-a4b4-b433ad6ac848"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:35.338230 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.338031 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-kube-api-access-22fbs" pod "2957d06d-fec2-40f0-a4b4-b433ad6ac848" (UID: "2957d06d-fec2-40f0-a4b4-b433ad6ac848"). InnerVolumeSpecName "kube-api-access-22fbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:35.436216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.436186 6287 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-cacerts\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:35.436216 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.436215 6287 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-ca-configmap\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:35.436419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.436226 6287 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2957d06d-fec2-40f0-a4b4-b433ad6ac848-local-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:35.436419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.436235 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fbs\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-kube-api-access-22fbs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:35.436419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.436244 6287 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-csr-dns-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:35.436419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.436254 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:35.436419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.436262 6287 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2957d06d-fec2-40f0-a4b4-b433ad6ac848-istio-kubeconfig\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:35.553557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.553465 6287 generic.go:356] "Generic (PLEG): container finished" podID="2957d06d-fec2-40f0-a4b4-b433ad6ac848" containerID="ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681" exitCode=0 Nov 01 22:50:35.553557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.553512 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" Nov 01 22:50:35.553557 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.553535 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" event={"ID":"2957d06d-fec2-40f0-a4b4-b433ad6ac848","Type":"ContainerDied","Data":"ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681"} Nov 01 22:50:35.553823 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.553584 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j" event={"ID":"2957d06d-fec2-40f0-a4b4-b433ad6ac848","Type":"ContainerDied","Data":"b1d234c5ee6dcbf0f2a47a4774bbab7c19334355283b3a045f86e25084c2cf80"} Nov 01 22:50:35.553823 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.553603 6287 scope.go:122] "RemoveContainer" containerID="ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681" Nov 01 22:50:35.564480 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.564453 6287 scope.go:122] "RemoveContainer" containerID="ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681" Nov 01 22:50:35.564746 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:50:35.564726 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681\": container with ID starting with ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681 not found: ID does not exist" containerID="ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681" Nov 01 22:50:35.564797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.564758 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681"} err="failed to get container status \"ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681\": rpc error: code = NotFound desc = could not find container \"ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681\": container with ID starting with ad818295ba2556d401171945a8d45e313ae72512af5750060615f3f609ee5681 not found: ID does not exist" Nov 01 22:50:35.596999 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.596963 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j"] Nov 01 22:50:35.616528 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:35.616488 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bjq2j"] Nov 01 22:50:36.623755 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:36.623718 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4"] Nov 01 22:50:36.635542 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:36.635503 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t"] Nov 01 22:50:37.020212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020173 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd"] Nov 01 22:50:37.020580 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020567 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" Nov 01 22:50:37.020645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020581 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" Nov 01 22:50:37.020645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020604 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2957d06d-fec2-40f0-a4b4-b433ad6ac848" containerName="discovery" Nov 01 22:50:37.020645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020609 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2957d06d-fec2-40f0-a4b4-b433ad6ac848" containerName="discovery" Nov 01 22:50:37.020645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020622 6287 cpu_manager.go:414] "containerMap: removing container" podUID="adb768ec-89c5-43f0-9c18-8d96741e91f1" containerName="revoke-keys" Nov 01 22:50:37.020645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020628 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb768ec-89c5-43f0-9c18-8d96741e91f1" containerName="revoke-keys" Nov 01 22:50:37.020645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020645 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" Nov 01 22:50:37.020842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020650 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" Nov 01 22:50:37.020842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020725 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb768ec-89c5-43f0-9c18-8d96741e91f1" containerName="revoke-keys" Nov 01 22:50:37.020842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020737 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" Nov 01 22:50:37.020842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020745 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" Nov 01 22:50:37.020842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020752 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b38708-4a75-40df-9835-d15d80301461" containerName="istio-proxy" Nov 01 22:50:37.020842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.020760 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2957d06d-fec2-40f0-a4b4-b433ad6ac848" containerName="discovery" Nov 01 22:50:37.025479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.025455 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.036827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.036799 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd"] Nov 01 22:50:37.122224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.122190 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:50:37.122379 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.122307 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:50:37.125373 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.125353 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:50:37.125471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.125393 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:50:37.152176 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.152137 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlt2\" (UniqueName: \"kubernetes.io/projected/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-kube-api-access-ltlt2\") pod \"maas-api-revoke-sub-1dd96b34-dhkrd\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.152309 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.152206 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd96b34-dhkrd\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.152355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.152299 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd96b34-dhkrd\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.252911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.252875 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlt2\" (UniqueName: \"kubernetes.io/projected/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-kube-api-access-ltlt2\") pod \"maas-api-revoke-sub-1dd96b34-dhkrd\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.253069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.252935 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd96b34-dhkrd\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.253069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.252968 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd96b34-dhkrd\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.253757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.253738 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd96b34-dhkrd\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.255435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.255408 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd96b34-dhkrd\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.261053 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.261028 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlt2\" (UniqueName: \"kubernetes.io/projected/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-kube-api-access-ltlt2\") pod \"maas-api-revoke-sub-1dd96b34-dhkrd\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.344611 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.344534 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:37.470032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.469998 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2957d06d-fec2-40f0-a4b4-b433ad6ac848" path="/var/lib/kubelet/pods/2957d06d-fec2-40f0-a4b4-b433ad6ac848/volumes" Nov 01 22:50:37.624519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:37.624492 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd"] Nov 01 22:50:37.626446 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:50:37.626424 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6c080e_a807_48f2_88c7_29ff8bce4cc5.slice/crio-38069743215df53ef83899138430dc082f19ba1b244f46e1a41ffe26b50d0a20 WatchSource:0}: Error finding container 38069743215df53ef83899138430dc082f19ba1b244f46e1a41ffe26b50d0a20: Status 404 returned error can't find the container with id 38069743215df53ef83899138430dc082f19ba1b244f46e1a41ffe26b50d0a20 Nov 01 22:50:38.576507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:38.576466 6287 generic.go:356] "Generic (PLEG): container finished" podID="dd6c080e-a807-48f2-88c7-29ff8bce4cc5" containerID="f8a213231796052121ec1515e9e40e4eede7f18e13b83debb9f043fe6577bc8f" exitCode=0 Nov 01 22:50:38.576659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:38.576527 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" event={"ID":"dd6c080e-a807-48f2-88c7-29ff8bce4cc5","Type":"ContainerDied","Data":"f8a213231796052121ec1515e9e40e4eede7f18e13b83debb9f043fe6577bc8f"} Nov 01 22:50:38.576659 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:38.576587 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" event={"ID":"dd6c080e-a807-48f2-88c7-29ff8bce4cc5","Type":"ContainerStarted","Data":"38069743215df53ef83899138430dc082f19ba1b244f46e1a41ffe26b50d0a20"} Nov 01 22:50:39.750479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.750455 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:39.882678 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.882600 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-xks-ca\") pod \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " Nov 01 22:50:39.882831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.882706 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-kube-api-access-ltlt2\" (UniqueName: \"kubernetes.io/projected/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-kube-api-access-ltlt2\") pod \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " Nov 01 22:50:39.882831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.882790 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-openshift-ca\") pod \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\" (UID: \"dd6c080e-a807-48f2-88c7-29ff8bce4cc5\") " Nov 01 22:50:39.883107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.883082 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-openshift-ca" pod "dd6c080e-a807-48f2-88c7-29ff8bce4cc5" (UID: "dd6c080e-a807-48f2-88c7-29ff8bce4cc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:39.884779 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.884751 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-kube-api-access-ltlt2" pod "dd6c080e-a807-48f2-88c7-29ff8bce4cc5" (UID: "dd6c080e-a807-48f2-88c7-29ff8bce4cc5"). InnerVolumeSpecName "kube-api-access-ltlt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:39.884875 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.884774 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-xks-ca" pod "dd6c080e-a807-48f2-88c7-29ff8bce4cc5" (UID: "dd6c080e-a807-48f2-88c7-29ff8bce4cc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:39.984079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.984046 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:39.984079 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.984076 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltlt2\" (UniqueName: \"kubernetes.io/projected/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-kube-api-access-ltlt2\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:39.984298 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:39.984087 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6c080e-a807-48f2-88c7-29ff8bce4cc5-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:40.594572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:40.594541 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" Nov 01 22:50:40.594746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:40.594575 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd" event={"ID":"dd6c080e-a807-48f2-88c7-29ff8bce4cc5","Type":"ContainerDied","Data":"38069743215df53ef83899138430dc082f19ba1b244f46e1a41ffe26b50d0a20"} Nov 01 22:50:40.594746 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:40.594652 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38069743215df53ef83899138430dc082f19ba1b244f46e1a41ffe26b50d0a20" Nov 01 22:50:41.624060 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:41.624003 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" podUID="230255df-b6dc-43d5-bcd7-364d2396ed06" containerName="payload-processing" containerID="cri-o://8d7714b871d4fd2a2ce06f9634f4c4499dd9f673db04a774c83a19eb1ef3dda1" gracePeriod=25 Nov 01 22:50:41.636555 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:41.636506 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" podUID="da7f0a84-0c8c-441e-8851-828d7d9949e8" containerName="payload-pre-processing" containerID="cri-o://50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415" gracePeriod=25 Nov 01 22:50:43.062206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:43.062171 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" podUID="da7f0a84-0c8c-441e-8851-828d7d9949e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:50:43.062206 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:43.062191 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" podUID="230255df-b6dc-43d5-bcd7-364d2396ed06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:50:45.589509 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:50:45.589473 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:50:46.308714 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.308644 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" containerID="cri-o://1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c" gracePeriod=45 Nov 01 22:50:46.329063 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.329031 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" containerID="cri-o://6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596" gracePeriod=45 Nov 01 22:50:46.333963 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.333917 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" containerID="cri-o://ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714" gracePeriod=45 Nov 01 22:50:46.595462 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.595077 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:50:46.633069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.633043 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:50:46.637610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.637584 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:50:46.639067 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.639041 6287 generic.go:356] "Generic (PLEG): container finished" podID="239a637b-db11-45f9-a13b-2739c43b99e5" containerID="6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596" exitCode=0 Nov 01 22:50:46.639240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.639098 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" Nov 01 22:50:46.639240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.639151 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" event={"ID":"239a637b-db11-45f9-a13b-2739c43b99e5","Type":"ContainerDied","Data":"6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596"} Nov 01 22:50:46.639240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.639191 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx" event={"ID":"239a637b-db11-45f9-a13b-2739c43b99e5","Type":"ContainerDied","Data":"355d8b05d6ef5466de3340c9912a60b5278ee6ebce4086d20e9ad6ee80da741d"} Nov 01 22:50:46.639240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.639215 6287 scope.go:122] "RemoveContainer" containerID="6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596" Nov 01 22:50:46.640898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.640853 6287 generic.go:356] "Generic (PLEG): container finished" podID="38090de6-418c-45f3-ae0f-e2905865c702" containerID="ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714" exitCode=0 Nov 01 22:50:46.641032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.640895 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" event={"ID":"38090de6-418c-45f3-ae0f-e2905865c702","Type":"ContainerDied","Data":"ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714"} Nov 01 22:50:46.641032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.640936 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" event={"ID":"38090de6-418c-45f3-ae0f-e2905865c702","Type":"ContainerDied","Data":"ce914506daf9ad6de89ffd54fbfac8f55d8c0b307dd563e43128458aa600a37b"} Nov 01 22:50:46.641032 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.640907 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs" Nov 01 22:50:46.642716 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.642695 6287 generic.go:356] "Generic (PLEG): container finished" podID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerID="1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c" exitCode=0 Nov 01 22:50:46.642829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.642741 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" Nov 01 22:50:46.642829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.642767 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" event={"ID":"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97","Type":"ContainerDied","Data":"1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c"} Nov 01 22:50:46.642829 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.642802 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5" event={"ID":"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97","Type":"ContainerDied","Data":"5baa8dc05dd964e78cce45a01c73d167cf328875efa7022b623e0e197401fe6b"} Nov 01 22:50:46.651948 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.651931 6287 scope.go:122] "RemoveContainer" containerID="6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596" Nov 01 22:50:46.652342 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:50:46.652318 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596\": container with ID starting with 6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596 not found: ID does not exist" containerID="6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596" Nov 01 22:50:46.652455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.652353 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596"} err="failed to get container status \"6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596\": rpc error: code = NotFound desc = could not find container \"6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596\": container with ID starting with 6f19cef92e983fac836f45fedf2b73c65710dbf99d58fd419949a3d7ce00e596 not found: ID does not exist" Nov 01 22:50:46.652455 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.652376 6287 scope.go:122] "RemoveContainer" containerID="ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714" Nov 01 22:50:46.671645 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.671621 6287 scope.go:122] "RemoveContainer" containerID="ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714" Nov 01 22:50:46.672084 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:50:46.672059 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714\": container with ID starting with ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714 not found: ID does not exist" containerID="ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714" Nov 01 22:50:46.672757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.672088 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714"} err="failed to get container status \"ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714\": rpc error: code = NotFound desc = could not find container \"ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714\": container with ID starting with ffb41467852592cce39476e8988ffd8568df246ca7bf81b3f10842f2bcdd1714 not found: ID does not exist" Nov 01 22:50:46.672757 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.672105 6287 scope.go:122] "RemoveContainer" containerID="1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c" Nov 01 22:50:46.690728 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.690703 6287 scope.go:122] "RemoveContainer" containerID="1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c" Nov 01 22:50:46.691294 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:50:46.691265 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c\": container with ID starting with 1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c not found: ID does not exist" containerID="1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c" Nov 01 22:50:46.691423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.691301 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c"} err="failed to get container status \"1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c\": rpc error: code = NotFound desc = could not find container \"1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c\": container with ID starting with 1efb5409a999ff0992114dc80c426580fa251e397e61784194b843af306e974c not found: ID does not exist" Nov 01 22:50:46.750440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.750409 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-dshm\") pod \"38090de6-418c-45f3-ae0f-e2905865c702\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " Nov 01 22:50:46.750636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.750447 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-dshm\") pod \"239a637b-db11-45f9-a13b-2739c43b99e5\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " Nov 01 22:50:46.750636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.750525 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-model-cache\") pod \"239a637b-db11-45f9-a13b-2739c43b99e5\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " Nov 01 22:50:46.750636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.750595 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tmp-dir\") pod \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " Nov 01 22:50:46.750636 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.750628 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tls-certs\") pod \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " Nov 01 22:50:46.750944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.750665 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-dshm\") pod \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " Nov 01 22:50:46.750944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.750722 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38090de6-418c-45f3-ae0f-e2905865c702-tls-certs\" (UniqueName: \"kubernetes.io/secret/38090de6-418c-45f3-ae0f-e2905865c702-tls-certs\") pod \"38090de6-418c-45f3-ae0f-e2905865c702\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " Nov 01 22:50:46.750944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.750900 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-model-cache" pod "239a637b-db11-45f9-a13b-2739c43b99e5" (UID: "239a637b-db11-45f9-a13b-2739c43b99e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.750944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.750926 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tmp-dir" pod "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" (UID: "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.751259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751141 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38090de6-418c-45f3-ae0f-e2905865c702-kube-api-access-sr5cq\" (UniqueName: \"kubernetes.io/projected/38090de6-418c-45f3-ae0f-e2905865c702-kube-api-access-sr5cq\") pod \"38090de6-418c-45f3-ae0f-e2905865c702\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " Nov 01 22:50:46.751259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751185 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/239a637b-db11-45f9-a13b-2739c43b99e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/239a637b-db11-45f9-a13b-2739c43b99e5-tls-certs\") pod \"239a637b-db11-45f9-a13b-2739c43b99e5\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " Nov 01 22:50:46.751259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751241 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-tmp-dir\") pod \"239a637b-db11-45f9-a13b-2739c43b99e5\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " Nov 01 22:50:46.751421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751283 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-model-cache\") pod \"38090de6-418c-45f3-ae0f-e2905865c702\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " Nov 01 22:50:46.751421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751346 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239a637b-db11-45f9-a13b-2739c43b99e5-kube-api-access-tn4nl\" (UniqueName: \"kubernetes.io/projected/239a637b-db11-45f9-a13b-2739c43b99e5-kube-api-access-tn4nl\") pod \"239a637b-db11-45f9-a13b-2739c43b99e5\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " Nov 01 22:50:46.751421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751398 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-model-cache\") pod \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " Nov 01 22:50:46.751596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751437 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-kube-api-access-nxbnk\" (UniqueName: \"kubernetes.io/projected/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-kube-api-access-nxbnk\") pod \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " Nov 01 22:50:46.751596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751465 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-home\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-home\") pod \"239a637b-db11-45f9-a13b-2739c43b99e5\" (UID: \"239a637b-db11-45f9-a13b-2739c43b99e5\") " Nov 01 22:50:46.751596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751507 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-tmp-dir\") pod \"38090de6-418c-45f3-ae0f-e2905865c702\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " Nov 01 22:50:46.751596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751537 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-home\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-home\") pod \"38090de6-418c-45f3-ae0f-e2905865c702\" (UID: \"38090de6-418c-45f3-ae0f-e2905865c702\") " Nov 01 22:50:46.751596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751574 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-home\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-home\") pod \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\" (UID: \"05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97\") " Nov 01 22:50:46.751876 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.751797 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-model-cache" pod "38090de6-418c-45f3-ae0f-e2905865c702" (UID: "38090de6-418c-45f3-ae0f-e2905865c702"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.752071 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.752045 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.752192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.752075 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.752192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.752092 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.753383 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.753356 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-dshm" pod "38090de6-418c-45f3-ae0f-e2905865c702" (UID: "38090de6-418c-45f3-ae0f-e2905865c702"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.753507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.753410 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-dshm" pod "239a637b-db11-45f9-a13b-2739c43b99e5" (UID: "239a637b-db11-45f9-a13b-2739c43b99e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.753507 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.753415 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tls-certs" pod "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" (UID: "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:46.753635 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.753605 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-tmp-dir" pod "239a637b-db11-45f9-a13b-2739c43b99e5" (UID: "239a637b-db11-45f9-a13b-2739c43b99e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.753704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.753656 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-model-cache" pod "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" (UID: "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.753704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.753675 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-home" pod "239a637b-db11-45f9-a13b-2739c43b99e5" (UID: "239a637b-db11-45f9-a13b-2739c43b99e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.753869 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.753847 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-home" pod "38090de6-418c-45f3-ae0f-e2905865c702" (UID: "38090de6-418c-45f3-ae0f-e2905865c702"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.753940 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.753927 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-home" pod "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" (UID: "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.754066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.754046 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-tmp-dir" pod "38090de6-418c-45f3-ae0f-e2905865c702" (UID: "38090de6-418c-45f3-ae0f-e2905865c702"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.754399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.754260 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38090de6-418c-45f3-ae0f-e2905865c702-kube-api-access-sr5cq" pod "38090de6-418c-45f3-ae0f-e2905865c702" (UID: "38090de6-418c-45f3-ae0f-e2905865c702"). InnerVolumeSpecName "kube-api-access-sr5cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:46.754399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.754385 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-dshm" pod "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" (UID: "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:46.754634 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.754610 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239a637b-db11-45f9-a13b-2739c43b99e5-kube-api-access-tn4nl" pod "239a637b-db11-45f9-a13b-2739c43b99e5" (UID: "239a637b-db11-45f9-a13b-2739c43b99e5"). InnerVolumeSpecName "kube-api-access-tn4nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:46.754793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.754772 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/239a637b-db11-45f9-a13b-2739c43b99e5-tls-certs" pod "239a637b-db11-45f9-a13b-2739c43b99e5" (UID: "239a637b-db11-45f9-a13b-2739c43b99e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:46.755084 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.755060 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38090de6-418c-45f3-ae0f-e2905865c702-tls-certs" pod "38090de6-418c-45f3-ae0f-e2905865c702" (UID: "38090de6-418c-45f3-ae0f-e2905865c702"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:50:46.755632 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.755606 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-kube-api-access-nxbnk" pod "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" (UID: "05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97"). InnerVolumeSpecName "kube-api-access-nxbnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:46.853362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853293 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr5cq\" (UniqueName: \"kubernetes.io/projected/38090de6-418c-45f3-ae0f-e2905865c702-kube-api-access-sr5cq\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853320 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/239a637b-db11-45f9-a13b-2739c43b99e5-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853329 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853337 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4nl\" (UniqueName: \"kubernetes.io/projected/239a637b-db11-45f9-a13b-2739c43b99e5-kube-api-access-tn4nl\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853349 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853357 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxbnk\" (UniqueName: \"kubernetes.io/projected/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-kube-api-access-nxbnk\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853362 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853366 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853375 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853383 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853390 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853397 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38090de6-418c-45f3-ae0f-e2905865c702-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853404 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/239a637b-db11-45f9-a13b-2739c43b99e5-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853412 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853419 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.853720 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.853426 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38090de6-418c-45f3-ae0f-e2905865c702-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:46.966589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.966565 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx"] Nov 01 22:50:46.969718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.969691 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/premium-w5-aa441b-kserve-66c7cbd7bb-m88cx"] Nov 01 22:50:46.982651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.982629 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs"] Nov 01 22:50:46.986737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.986716 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/simulator-w5-aa441b-kserve-54b574d455-n7qrs"] Nov 01 22:50:46.996445 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.996415 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5"] Nov 01 22:50:46.998969 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:46.998949 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-aa441b/distinct-2-w5-aa441b-kserve-554cd9786-sdmc5"] Nov 01 22:50:47.470414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:47.470384 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" path="/var/lib/kubelet/pods/05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97/volumes" Nov 01 22:50:47.470793 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:47.470780 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" path="/var/lib/kubelet/pods/239a637b-db11-45f9-a13b-2739c43b99e5/volumes" Nov 01 22:50:47.471166 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:47.471153 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38090de6-418c-45f3-ae0f-e2905865c702" path="/var/lib/kubelet/pods/38090de6-418c-45f3-ae0f-e2905865c702/volumes" Nov 01 22:50:53.062205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:53.062167 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" podUID="da7f0a84-0c8c-441e-8851-828d7d9949e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:50:53.062621 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:53.062175 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" podUID="230255df-b6dc-43d5-bcd7-364d2396ed06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:50:59.502812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:59.502763 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b"] Nov 01 22:50:59.503377 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:59.503038 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" containerID="cri-o://c2dd10aacbab53cde0f36febcbbdbb1ba36a10cc335425dac98c031eecfc7659" gracePeriod=30 Nov 01 22:50:59.520856 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:59.520825 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw"] Nov 01 22:50:59.521056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:50:59.521029 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" containerID="cri-o://646b3135bf874c4ad6b772157a35e920daa0c47a7fa16c94c322fbd72615e44f" gracePeriod=30 Nov 01 22:51:01.332532 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:01.332499 6287 patch_prober.go:29] interesting pod/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:51:01.332936 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:01.332570 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:51:01.751597 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:01.751566 6287 generic.go:356] "Generic (PLEG): container finished" podID="230255df-b6dc-43d5-bcd7-364d2396ed06" containerID="8d7714b871d4fd2a2ce06f9634f4c4499dd9f673db04a774c83a19eb1ef3dda1" exitCode=0 Nov 01 22:51:01.751700 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:01.751639 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" event={"ID":"230255df-b6dc-43d5-bcd7-364d2396ed06","Type":"ContainerDied","Data":"8d7714b871d4fd2a2ce06f9634f4c4499dd9f673db04a774c83a19eb1ef3dda1"} Nov 01 22:51:01.880226 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:01.880172 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:51:01.925066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:01.925032 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:51:02.002919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.002874 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230255df-b6dc-43d5-bcd7-364d2396ed06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230255df-b6dc-43d5-bcd7-364d2396ed06-plugins-config-volume\") pod \"230255df-b6dc-43d5-bcd7-364d2396ed06\" (UID: \"230255df-b6dc-43d5-bcd7-364d2396ed06\") " Nov 01 22:51:02.003140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.002934 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7f0a84-0c8c-441e-8851-828d7d9949e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7f0a84-0c8c-441e-8851-828d7d9949e8-plugins-config-volume\") pod \"da7f0a84-0c8c-441e-8851-828d7d9949e8\" (UID: \"da7f0a84-0c8c-441e-8851-828d7d9949e8\") " Nov 01 22:51:02.003140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.002988 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7f0a84-0c8c-441e-8851-828d7d9949e8-kube-api-access-v799r\" (UniqueName: \"kubernetes.io/projected/da7f0a84-0c8c-441e-8851-828d7d9949e8-kube-api-access-v799r\") pod \"da7f0a84-0c8c-441e-8851-828d7d9949e8\" (UID: \"da7f0a84-0c8c-441e-8851-828d7d9949e8\") " Nov 01 22:51:02.003140 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.003044 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230255df-b6dc-43d5-bcd7-364d2396ed06-kube-api-access-gdgn6\" (UniqueName: \"kubernetes.io/projected/230255df-b6dc-43d5-bcd7-364d2396ed06-kube-api-access-gdgn6\") pod \"230255df-b6dc-43d5-bcd7-364d2396ed06\" (UID: \"230255df-b6dc-43d5-bcd7-364d2396ed06\") " Nov 01 22:51:02.003419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.003391 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230255df-b6dc-43d5-bcd7-364d2396ed06-plugins-config-volume" pod "230255df-b6dc-43d5-bcd7-364d2396ed06" (UID: "230255df-b6dc-43d5-bcd7-364d2396ed06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:51:02.003485 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.003407 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7f0a84-0c8c-441e-8851-828d7d9949e8-plugins-config-volume" pod "da7f0a84-0c8c-441e-8851-828d7d9949e8" (UID: "da7f0a84-0c8c-441e-8851-828d7d9949e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:51:02.005284 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.005256 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230255df-b6dc-43d5-bcd7-364d2396ed06-kube-api-access-gdgn6" pod "230255df-b6dc-43d5-bcd7-364d2396ed06" (UID: "230255df-b6dc-43d5-bcd7-364d2396ed06"). InnerVolumeSpecName "kube-api-access-gdgn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:02.005428 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.005410 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7f0a84-0c8c-441e-8851-828d7d9949e8-kube-api-access-v799r" pod "da7f0a84-0c8c-441e-8851-828d7d9949e8" (UID: "da7f0a84-0c8c-441e-8851-828d7d9949e8"). InnerVolumeSpecName "kube-api-access-v799r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:02.105004 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.104961 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdgn6\" (UniqueName: \"kubernetes.io/projected/230255df-b6dc-43d5-bcd7-364d2396ed06-kube-api-access-gdgn6\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:02.105004 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.105002 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230255df-b6dc-43d5-bcd7-364d2396ed06-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:02.105224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.105015 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7f0a84-0c8c-441e-8851-828d7d9949e8-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:02.105224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.105027 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v799r\" (UniqueName: \"kubernetes.io/projected/da7f0a84-0c8c-441e-8851-828d7d9949e8-kube-api-access-v799r\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:02.760486 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.760455 6287 generic.go:356] "Generic (PLEG): container finished" podID="da7f0a84-0c8c-441e-8851-828d7d9949e8" containerID="50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415" exitCode=0 Nov 01 22:51:02.760947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.760502 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" Nov 01 22:51:02.760947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.760545 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" event={"ID":"da7f0a84-0c8c-441e-8851-828d7d9949e8","Type":"ContainerDied","Data":"50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415"} Nov 01 22:51:02.760947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.760606 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t" event={"ID":"da7f0a84-0c8c-441e-8851-828d7d9949e8","Type":"ContainerDied","Data":"fe232bbe78e91f2cb376f5c4aa452fad8e5d7bcf9f928aca4fdeaff89373ec10"} Nov 01 22:51:02.760947 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.760621 6287 scope.go:122] "RemoveContainer" containerID="50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415" Nov 01 22:51:02.762011 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.761990 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" event={"ID":"230255df-b6dc-43d5-bcd7-364d2396ed06","Type":"ContainerDied","Data":"6918b00a88d16cee7f2d8eaf2ba97701f150e0781afd2a9d63972e574a5f7a7d"} Nov 01 22:51:02.762091 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.762012 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4" Nov 01 22:51:02.771741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.771719 6287 scope.go:122] "RemoveContainer" containerID="50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415" Nov 01 22:51:02.772036 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:51:02.772011 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415\": container with ID starting with 50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415 not found: ID does not exist" containerID="50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415" Nov 01 22:51:02.772163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.772048 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415"} err="failed to get container status \"50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415\": rpc error: code = NotFound desc = could not find container \"50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415\": container with ID starting with 50d01c5298fc27e0765df8e6e51564cd29cc0785aabc4939952c9b05b1dd5415 not found: ID does not exist" Nov 01 22:51:02.772163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.772070 6287 scope.go:122] "RemoveContainer" containerID="8d7714b871d4fd2a2ce06f9634f4c4499dd9f673db04a774c83a19eb1ef3dda1" Nov 01 22:51:02.789894 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.789862 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4"] Nov 01 22:51:02.793571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.793548 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-aa441b-7d989db8cf-5knv4"] Nov 01 22:51:02.808075 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.808049 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t"] Nov 01 22:51:02.817701 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:02.817682 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-aa441b-764764f459-5vm6t"] Nov 01 22:51:03.470299 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:03.470261 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230255df-b6dc-43d5-bcd7-364d2396ed06" path="/var/lib/kubelet/pods/230255df-b6dc-43d5-bcd7-364d2396ed06/volumes" Nov 01 22:51:03.470863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:03.470836 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7f0a84-0c8c-441e-8851-828d7d9949e8" path="/var/lib/kubelet/pods/da7f0a84-0c8c-441e-8851-828d7d9949e8/volumes" Nov 01 22:51:04.857147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:04.851730 6287 generic.go:356] "Generic (PLEG): container finished" podID="c68acbcb-66c9-4def-af31-184d7544248d" containerID="c2dd10aacbab53cde0f36febcbbdbb1ba36a10cc335425dac98c031eecfc7659" exitCode=0 Nov 01 22:51:04.857147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:04.851781 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" event={"ID":"c68acbcb-66c9-4def-af31-184d7544248d","Type":"ContainerDied","Data":"c2dd10aacbab53cde0f36febcbbdbb1ba36a10cc335425dac98c031eecfc7659"} Nov 01 22:51:04.857147 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:04.851984 6287 scope.go:122] "RemoveContainer" containerID="4c9554c4500e6f2bf3967e64df2314825d4f45c1388bb1fec9737e2c3e42adc0" Nov 01 22:51:04.858249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:04.857200 6287 generic.go:356] "Generic (PLEG): container finished" podID="bad51f46-a415-4fad-9071-8ece456756e9" containerID="646b3135bf874c4ad6b772157a35e920daa0c47a7fa16c94c322fbd72615e44f" exitCode=0 Nov 01 22:51:04.858249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:04.857237 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" event={"ID":"bad51f46-a415-4fad-9071-8ece456756e9","Type":"ContainerDied","Data":"646b3135bf874c4ad6b772157a35e920daa0c47a7fa16c94c322fbd72615e44f"} Nov 01 22:51:05.066771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.066701 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:51:05.158069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.157009 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-istio-token\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-istio-token\") pod \"bad51f46-a415-4fad-9071-8ece456756e9\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " Nov 01 22:51:05.158069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.157079 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-socket\") pod \"bad51f46-a415-4fad-9071-8ece456756e9\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " Nov 01 22:51:05.158069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.157128 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bad51f46-a415-4fad-9071-8ece456756e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad51f46-a415-4fad-9071-8ece456756e9-istio-podinfo\") pod \"bad51f46-a415-4fad-9071-8ece456756e9\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " Nov 01 22:51:05.158069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.157198 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-credential-socket\") pod \"bad51f46-a415-4fad-9071-8ece456756e9\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " Nov 01 22:51:05.158069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.157249 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-envoy\") pod \"bad51f46-a415-4fad-9071-8ece456756e9\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " Nov 01 22:51:05.158069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.157329 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-certs\") pod \"bad51f46-a415-4fad-9071-8ece456756e9\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " Nov 01 22:51:05.158069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.157408 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-kube-api-access-pbmmp\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-kube-api-access-pbmmp\") pod \"bad51f46-a415-4fad-9071-8ece456756e9\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " Nov 01 22:51:05.158069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.157444 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-data\") pod \"bad51f46-a415-4fad-9071-8ece456756e9\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " Nov 01 22:51:05.158069 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.157474 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad51f46-a415-4fad-9071-8ece456756e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad51f46-a415-4fad-9071-8ece456756e9-istiod-ca-cert\") pod \"bad51f46-a415-4fad-9071-8ece456756e9\" (UID: \"bad51f46-a415-4fad-9071-8ece456756e9\") " Nov 01 22:51:05.158803 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.158761 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-certs" pod "bad51f46-a415-4fad-9071-8ece456756e9" (UID: "bad51f46-a415-4fad-9071-8ece456756e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.159086 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.158993 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-credential-socket" pod "bad51f46-a415-4fad-9071-8ece456756e9" (UID: "bad51f46-a415-4fad-9071-8ece456756e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.159086 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.159030 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad51f46-a415-4fad-9071-8ece456756e9-istiod-ca-cert" pod "bad51f46-a415-4fad-9071-8ece456756e9" (UID: "bad51f46-a415-4fad-9071-8ece456756e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:51:05.161438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.161412 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-data" pod "bad51f46-a415-4fad-9071-8ece456756e9" (UID: "bad51f46-a415-4fad-9071-8ece456756e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.161589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.161563 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-socket" pod "bad51f46-a415-4fad-9071-8ece456756e9" (UID: "bad51f46-a415-4fad-9071-8ece456756e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.161878 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.161852 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bad51f46-a415-4fad-9071-8ece456756e9-istio-podinfo" pod "bad51f46-a415-4fad-9071-8ece456756e9" (UID: "bad51f46-a415-4fad-9071-8ece456756e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:51:05.164534 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.164509 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-envoy" pod "bad51f46-a415-4fad-9071-8ece456756e9" (UID: "bad51f46-a415-4fad-9071-8ece456756e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.164630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.164545 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-kube-api-access-pbmmp" pod "bad51f46-a415-4fad-9071-8ece456756e9" (UID: "bad51f46-a415-4fad-9071-8ece456756e9"). InnerVolumeSpecName "kube-api-access-pbmmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:05.164685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.164671 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-istio-token" pod "bad51f46-a415-4fad-9071-8ece456756e9" (UID: "bad51f46-a415-4fad-9071-8ece456756e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:05.165799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.165778 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:51:05.258162 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258094 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-envoy\") pod \"c68acbcb-66c9-4def-af31-184d7544248d\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " Nov 01 22:51:05.258334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258201 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c68acbcb-66c9-4def-af31-184d7544248d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68acbcb-66c9-4def-af31-184d7544248d-istio-podinfo\") pod \"c68acbcb-66c9-4def-af31-184d7544248d\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " Nov 01 22:51:05.258334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258236 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-data\") pod \"c68acbcb-66c9-4def-af31-184d7544248d\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " Nov 01 22:51:05.258334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258278 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-certs\") pod \"c68acbcb-66c9-4def-af31-184d7544248d\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " Nov 01 22:51:05.258334 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258307 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68acbcb-66c9-4def-af31-184d7544248d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68acbcb-66c9-4def-af31-184d7544248d-istiod-ca-cert\") pod \"c68acbcb-66c9-4def-af31-184d7544248d\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " Nov 01 22:51:05.258479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258338 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-istio-token\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-istio-token\") pod \"c68acbcb-66c9-4def-af31-184d7544248d\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " Nov 01 22:51:05.258479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258367 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-kube-api-access-5wrqh\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-kube-api-access-5wrqh\") pod \"c68acbcb-66c9-4def-af31-184d7544248d\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " Nov 01 22:51:05.258479 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258455 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-credential-socket\") pod \"c68acbcb-66c9-4def-af31-184d7544248d\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " Nov 01 22:51:05.258584 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258499 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-socket\") pod \"c68acbcb-66c9-4def-af31-184d7544248d\" (UID: \"c68acbcb-66c9-4def-af31-184d7544248d\") " Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258895 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258924 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbmmp\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-kube-api-access-pbmmp\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258940 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258953 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad51f46-a415-4fad-9071-8ece456756e9-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258967 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad51f46-a415-4fad-9071-8ece456756e9-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258980 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.258995 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad51f46-a415-4fad-9071-8ece456756e9-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.259009 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.259024 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad51f46-a415-4fad-9071-8ece456756e9-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.259183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.259048 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-certs" pod "c68acbcb-66c9-4def-af31-184d7544248d" (UID: "c68acbcb-66c9-4def-af31-184d7544248d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.260238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.260211 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68acbcb-66c9-4def-af31-184d7544248d-istiod-ca-cert" pod "c68acbcb-66c9-4def-af31-184d7544248d" (UID: "c68acbcb-66c9-4def-af31-184d7544248d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:51:05.260312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.260236 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-data" pod "c68acbcb-66c9-4def-af31-184d7544248d" (UID: "c68acbcb-66c9-4def-af31-184d7544248d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.260312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.260257 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-socket" pod "c68acbcb-66c9-4def-af31-184d7544248d" (UID: "c68acbcb-66c9-4def-af31-184d7544248d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.260653 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.260626 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-credential-socket" pod "c68acbcb-66c9-4def-af31-184d7544248d" (UID: "c68acbcb-66c9-4def-af31-184d7544248d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.262477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.262452 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-istio-token" pod "c68acbcb-66c9-4def-af31-184d7544248d" (UID: "c68acbcb-66c9-4def-af31-184d7544248d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:05.262559 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.262489 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c68acbcb-66c9-4def-af31-184d7544248d-istio-podinfo" pod "c68acbcb-66c9-4def-af31-184d7544248d" (UID: "c68acbcb-66c9-4def-af31-184d7544248d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:51:05.262602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.262561 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-envoy" pod "c68acbcb-66c9-4def-af31-184d7544248d" (UID: "c68acbcb-66c9-4def-af31-184d7544248d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:51:05.262602 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.262575 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-kube-api-access-5wrqh" pod "c68acbcb-66c9-4def-af31-184d7544248d" (UID: "c68acbcb-66c9-4def-af31-184d7544248d"). InnerVolumeSpecName "kube-api-access-5wrqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:51:05.360208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.360094 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.360208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.360170 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68acbcb-66c9-4def-af31-184d7544248d-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.360208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.360187 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.360208 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.360202 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.360465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.360216 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68acbcb-66c9-4def-af31-184d7544248d-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.360465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.360231 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.360465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.360245 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wrqh\" (UniqueName: \"kubernetes.io/projected/c68acbcb-66c9-4def-af31-184d7544248d-kube-api-access-5wrqh\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.360465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.360261 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.360465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.360274 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68acbcb-66c9-4def-af31-184d7544248d-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:51:05.864711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.864681 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" Nov 01 22:51:05.865205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.864719 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw" event={"ID":"bad51f46-a415-4fad-9071-8ece456756e9","Type":"ContainerDied","Data":"5f1cfe9cfc94085d40034e45c2e54d531d7fe33dae25db8a10795264f04acb3d"} Nov 01 22:51:05.865205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.864779 6287 scope.go:122] "RemoveContainer" containerID="646b3135bf874c4ad6b772157a35e920daa0c47a7fa16c94c322fbd72615e44f" Nov 01 22:51:05.866539 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.866509 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" Nov 01 22:51:05.866663 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.866533 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b" event={"ID":"c68acbcb-66c9-4def-af31-184d7544248d","Type":"ContainerDied","Data":"70ffe4be9f5e03fc99a5e259445e73176a030d6144ae1a729f1205a4d2190143"} Nov 01 22:51:05.876240 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.876222 6287 scope.go:122] "RemoveContainer" containerID="c2dd10aacbab53cde0f36febcbbdbb1ba36a10cc335425dac98c031eecfc7659" Nov 01 22:51:05.883840 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.883816 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw"] Nov 01 22:51:05.890246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.890225 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-mkjpw"] Nov 01 22:51:05.903080 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.903059 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b"] Nov 01 22:51:05.905477 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:05.905459 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-aa441b-openshift-default-59698dbb8f-26t7b"] Nov 01 22:51:07.471106 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:07.471072 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad51f46-a415-4fad-9071-8ece456756e9" path="/var/lib/kubelet/pods/bad51f46-a415-4fad-9071-8ece456756e9/volumes" Nov 01 22:51:07.471550 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:07.471535 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68acbcb-66c9-4def-af31-184d7544248d" path="/var/lib/kubelet/pods/c68acbcb-66c9-4def-af31-184d7544248d/volumes" Nov 01 22:51:15.586534 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:51:15.586493 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:51:31.192496 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:31.192466 6287 scope.go:122] "RemoveContainer" containerID="2883be8cced56a999ccd07f760edfc8aacf84eeadd06007478b4384d4c6b0103" Nov 01 22:51:31.201704 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:31.201680 6287 scope.go:122] "RemoveContainer" containerID="29f95c319ac41c4515c5ae21d31580658f60c9fe8b01388da54f4c81b1ebe209" Nov 01 22:51:31.210977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:31.210958 6287 scope.go:122] "RemoveContainer" containerID="5d49a0ea37b9657868e137ea708b72bfcc3857def875627a49b462c3c3d978e4" Nov 01 22:51:31.219375 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:31.219360 6287 scope.go:122] "RemoveContainer" containerID="d618757e2bb869b479653daeee40c2cb707efde127d66883f0fa462983888f6e" Nov 01 22:51:31.227896 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:31.227882 6287 scope.go:122] "RemoveContainer" containerID="fca402f8081c9033150942ec78e8d6633d233b4596aa8bd72416c5f660b8dcb9" Nov 01 22:51:31.235883 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:31.235867 6287 scope.go:122] "RemoveContainer" containerID="8f4155aca1c574d1d0df78abad1b649cae21491072c9b4c8c90af40b87d14771" Nov 01 22:51:31.243710 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:31.243691 6287 scope.go:122] "RemoveContainer" containerID="2f52bcbf02ed5c0c4f40ac996e4342f6ee990c58fe4fced0fa5dc35253c89b5e" Nov 01 22:51:31.251642 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:31.251624 6287 scope.go:122] "RemoveContainer" containerID="b463a29e3fcb46c4f2c3baeb9991f73bac22af06b6d4bf9602d61d16337c92ad" Nov 01 22:51:31.259355 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:31.259337 6287 scope.go:122] "RemoveContainer" containerID="2fcb93bffadb7cac2617b5951c8f85635983d84ead9fac79692412e2d0ac514e" Nov 01 22:51:32.021316 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:32.021279 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t"] Nov 01 22:51:32.024629 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:32.024600 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt"] Nov 01 22:51:32.027723 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:32.027701 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb73e4df-zn49t"] Nov 01 22:51:32.030553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:32.030530 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e1b140d-5pqzt"] Nov 01 22:51:33.469600 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:33.469563 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5a4fb8-2c34-4d00-9c14-f94e8547c731" path="/var/lib/kubelet/pods/2d5a4fb8-2c34-4d00-9c14-f94e8547c731/volumes" Nov 01 22:51:33.470184 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:51:33.470165 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f595b3e9-8ea8-4f90-b79c-5532828aacd3" path="/var/lib/kubelet/pods/f595b3e9-8ea8-4f90-b79c-5532828aacd3/volumes" Nov 01 22:51:45.588060 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:51:45.588020 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:52:01.023881 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.023842 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz"] Nov 01 22:52:01.027644 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.027616 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98"] Nov 01 22:52:01.033424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.033402 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e44f43a-lh4hz"] Nov 01 22:52:01.036093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.036068 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz"] Nov 01 22:52:01.037986 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.037967 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx"] Nov 01 22:52:01.040066 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.040043 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed5a44d-c8j98"] Nov 01 22:52:01.041703 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.041679 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d96147-98drz"] Nov 01 22:52:01.043354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.043325 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd229ef-gg9gx"] Nov 01 22:52:01.469831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.469756 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b1e2363-75e0-4edf-9171-d9166d9a340d" path="/var/lib/kubelet/pods/6b1e2363-75e0-4edf-9171-d9166d9a340d/volumes" Nov 01 22:52:01.470331 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.470314 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1" path="/var/lib/kubelet/pods/a2a617d4-a2d4-45de-bf0b-4a54d07b8fe1/volumes" Nov 01 22:52:01.470682 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.470669 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ef43af-1174-4cff-8176-72e91a86d6bf" path="/var/lib/kubelet/pods/b0ef43af-1174-4cff-8176-72e91a86d6bf/volumes" Nov 01 22:52:01.471030 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:01.471018 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db7cbe18-b450-4519-8ce9-df81d1a3e943" path="/var/lib/kubelet/pods/db7cbe18-b450-4519-8ce9-df81d1a3e943/volumes" Nov 01 22:52:15.587036 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:52:15.586998 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:52:31.360919 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:31.360885 6287 scope.go:122] "RemoveContainer" containerID="eca48feed66cf571235fca83cd3fe383e49fbeecc80e4ff3bd14a812bd925531" Nov 01 22:52:31.370057 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:31.370037 6287 scope.go:122] "RemoveContainer" containerID="c2ba21b6add30c93bc81b24eade133c650c4a0e591b47b11769d75fb2d2a7cbb" Nov 01 22:52:31.437134 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:31.433248 6287 scope.go:122] "RemoveContainer" containerID="3591442eb1fdc80d1f3f5a18c4812721c3921ed03de42100cefcb908eac28774" Nov 01 22:52:31.450451 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:31.450408 6287 scope.go:122] "RemoveContainer" containerID="b76d68d10cc05616b22e285048114bf74c7c533d18920e82ee04e943c90565d2" Nov 01 22:52:31.458827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:31.458803 6287 scope.go:122] "RemoveContainer" containerID="31e9f9b89b502e08685ab55d826c7460a4fad6e044109c2115704f15c057ae77" Nov 01 22:52:31.467325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:31.467307 6287 scope.go:122] "RemoveContainer" containerID="6de786ff767c59344343ae29e19919cdac49bd343512c12e420bf864d38b813d" Nov 01 22:52:31.553827 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:31.553769 6287 scope.go:122] "RemoveContainer" containerID="6c88108c4b0cf6aa44c70e08aeedc0388bf182cde25373fd84fada58110d40cb" Nov 01 22:52:31.562135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:31.562098 6287 scope.go:122] "RemoveContainer" containerID="2a961ae10f0630ffde89a5adcbb9d5582bc18929d650784b54b09968f5396aff" Nov 01 22:52:31.600137 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:31.598315 6287 scope.go:122] "RemoveContainer" containerID="7997d1aaabfd6c03603a56192afc17728c2a3c4860c13ca7ee386d8c64a7fa71" Nov 01 22:52:33.030921 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:33.030887 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66"] Nov 01 22:52:33.036483 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:33.036457 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s"] Nov 01 22:52:33.043327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:33.043301 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b132126-nkn2s"] Nov 01 22:52:33.053898 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:33.053870 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b706d81-5mk66"] Nov 01 22:52:33.469811 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:33.469734 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6" path="/var/lib/kubelet/pods/bab9a3cf-0ee0-4cbc-9a55-0f1c81996fb6/volumes" Nov 01 22:52:33.470148 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:52:33.470133 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2dd6b25-84e7-4518-b112-3ad691d96e98" path="/var/lib/kubelet/pods/e2dd6b25-84e7-4518-b112-3ad691d96e98/volumes" Nov 01 22:52:45.589732 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:52:45.589691 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:53:15.586642 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:53:15.586603 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:53:17.271332 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:17.271265 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 22:53:17.271930 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:17.271912 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 22:53:19.700174 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:19.700149 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 22:53:19.700849 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:19.700828 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 22:53:20.272563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.271746 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9"] Nov 01 22:53:20.272563 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.272084 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" podUID="f08dabd9-50ad-49a9-ba37-70a59b58c8c4" containerName="discovery" containerID="cri-o://fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87" gracePeriod=30 Nov 01 22:53:20.524658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.524603 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:53:20.653061 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.653031 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-dns-cert\") pod \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " Nov 01 22:53:20.653259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.653075 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-kube-api-access-cdxgz\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-kube-api-access-cdxgz\") pod \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " Nov 01 22:53:20.653259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.653103 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-local-certs\") pod \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " Nov 01 22:53:20.653259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.653154 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-cacerts\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-cacerts\") pod \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " Nov 01 22:53:20.653259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.653181 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-kubeconfig\") pod \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " Nov 01 22:53:20.653259 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.653219 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-ca-configmap\") pod \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " Nov 01 22:53:20.653546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.653267 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-token\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-token\") pod \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\" (UID: \"f08dabd9-50ad-49a9-ba37-70a59b58c8c4\") " Nov 01 22:53:20.653715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.653677 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-ca-configmap" pod "f08dabd9-50ad-49a9-ba37-70a59b58c8c4" (UID: "f08dabd9-50ad-49a9-ba37-70a59b58c8c4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:53:20.655679 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.655645 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-dns-cert" pod "f08dabd9-50ad-49a9-ba37-70a59b58c8c4" (UID: "f08dabd9-50ad-49a9-ba37-70a59b58c8c4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:53:20.655798 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.655653 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-token" pod "f08dabd9-50ad-49a9-ba37-70a59b58c8c4" (UID: "f08dabd9-50ad-49a9-ba37-70a59b58c8c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:53:20.655855 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.655835 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-local-certs" pod "f08dabd9-50ad-49a9-ba37-70a59b58c8c4" (UID: "f08dabd9-50ad-49a9-ba37-70a59b58c8c4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:20.655991 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.655967 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-cacerts" pod "f08dabd9-50ad-49a9-ba37-70a59b58c8c4" (UID: "f08dabd9-50ad-49a9-ba37-70a59b58c8c4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:53:20.656056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.655980 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-kubeconfig" pod "f08dabd9-50ad-49a9-ba37-70a59b58c8c4" (UID: "f08dabd9-50ad-49a9-ba37-70a59b58c8c4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:53:20.656056 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.655994 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-kube-api-access-cdxgz" pod "f08dabd9-50ad-49a9-ba37-70a59b58c8c4" (UID: "f08dabd9-50ad-49a9-ba37-70a59b58c8c4"). InnerVolumeSpecName "kube-api-access-cdxgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:53:20.754274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.754239 6287 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-dns-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:20.754274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.754268 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdxgz\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-kube-api-access-cdxgz\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:20.754274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.754279 6287 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-local-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:20.754718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.754289 6287 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-cacerts\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:20.754718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.754299 6287 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-kubeconfig\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:20.754718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.754308 6287 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-csr-ca-configmap\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:20.754718 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.754316 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f08dabd9-50ad-49a9-ba37-70a59b58c8c4-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:20.855327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.855241 6287 generic.go:356] "Generic (PLEG): container finished" podID="f08dabd9-50ad-49a9-ba37-70a59b58c8c4" containerID="fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87" exitCode=0 Nov 01 22:53:20.855327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.855285 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" event={"ID":"f08dabd9-50ad-49a9-ba37-70a59b58c8c4","Type":"ContainerDied","Data":"fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87"} Nov 01 22:53:20.855327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.855290 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" Nov 01 22:53:20.855327 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.855321 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9" event={"ID":"f08dabd9-50ad-49a9-ba37-70a59b58c8c4","Type":"ContainerDied","Data":"9425269bad321aff725101d2110e48241fd44dd7f46939ec4f1ea80e0edce9dd"} Nov 01 22:53:20.855670 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.855341 6287 scope.go:122] "RemoveContainer" containerID="fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87" Nov 01 22:53:20.869431 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.869406 6287 scope.go:122] "RemoveContainer" containerID="fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87" Nov 01 22:53:20.869753 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:53:20.869732 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87\": container with ID starting with fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87 not found: ID does not exist" containerID="fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87" Nov 01 22:53:20.869808 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.869761 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87"} err="failed to get container status \"fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87\": rpc error: code = NotFound desc = could not find container \"fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87\": container with ID starting with fcb3406550d84f07a26576c006b87a9eb7bdb10a3b78c34f53aa09be11f5fe87 not found: ID does not exist" Nov 01 22:53:20.877577 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.877549 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9"] Nov 01 22:53:20.882771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:20.882747 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9nqs9"] Nov 01 22:53:21.470107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:21.470071 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08dabd9-50ad-49a9-ba37-70a59b58c8c4" path="/var/lib/kubelet/pods/f08dabd9-50ad-49a9-ba37-70a59b58c8c4/volumes" Nov 01 22:53:22.506435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506402 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s"] Nov 01 22:53:22.506828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506762 6287 cpu_manager.go:414] "containerMap: removing container" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" Nov 01 22:53:22.506828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506776 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" Nov 01 22:53:22.506828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506797 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" Nov 01 22:53:22.506828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506802 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f18200f-e7cd-4aa5-a2b8-3d81b84b0186" containerName="istio-proxy" Nov 01 22:53:22.506828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506812 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dd6c080e-a807-48f2-88c7-29ff8bce4cc5" containerName="revoke-keys" Nov 01 22:53:22.506828 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506817 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6c080e-a807-48f2-88c7-29ff8bce4cc5" containerName="revoke-keys" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506844 6287 cpu_manager.go:414] "containerMap: removing container" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506850 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506859 6287 cpu_manager.go:414] "containerMap: removing container" podUID="230255df-b6dc-43d5-bcd7-364d2396ed06" containerName="payload-processing" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506864 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230255df-b6dc-43d5-bcd7-364d2396ed06" containerName="payload-processing" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506880 6287 cpu_manager.go:414] "containerMap: removing container" podUID="da7f0a84-0c8c-441e-8851-828d7d9949e8" containerName="payload-pre-processing" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506886 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7f0a84-0c8c-441e-8851-828d7d9949e8" containerName="payload-pre-processing" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506896 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506902 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506911 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f08dabd9-50ad-49a9-ba37-70a59b58c8c4" containerName="discovery" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506916 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08dabd9-50ad-49a9-ba37-70a59b58c8c4" containerName="discovery" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506925 6287 cpu_manager.go:414] "containerMap: removing container" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506929 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506937 6287 cpu_manager.go:414] "containerMap: removing container" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506942 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506948 6287 cpu_manager.go:414] "containerMap: removing container" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.506952 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507008 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="38090de6-418c-45f3-ae0f-e2905865c702" containerName="main" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507017 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="230255df-b6dc-43d5-bcd7-364d2396ed06" containerName="payload-processing" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507023 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7f0a84-0c8c-441e-8851-828d7d9949e8" containerName="payload-pre-processing" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507031 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507037 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08dabd9-50ad-49a9-ba37-70a59b58c8c4" containerName="discovery" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507044 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="239a637b-db11-45f9-a13b-2739c43b99e5" containerName="main" Nov 01 22:53:22.507041 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507050 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68acbcb-66c9-4def-af31-184d7544248d" containerName="istio-proxy" Nov 01 22:53:22.507770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507055 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad51f46-a415-4fad-9071-8ece456756e9" containerName="istio-proxy" Nov 01 22:53:22.507770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507061 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a6224f-c5a4-4ed1-8cbb-b1b4216a6f97" containerName="main" Nov 01 22:53:22.507770 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.507067 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6c080e-a807-48f2-88c7-29ff8bce4cc5" containerName="revoke-keys" Nov 01 22:53:22.511648 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.511629 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.516669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.516641 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:53:22.517444 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.517423 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:53:22.517603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.517504 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:53:22.517719 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.517571 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:53:22.520106 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.520083 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s"] Nov 01 22:53:22.669183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.669150 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ckv8s\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.669361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.669207 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ckv8s\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.669361 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.669318 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wdq\" (UniqueName: \"kubernetes.io/projected/20ab58dc-1516-416e-b0d8-835301e9f0e2-kube-api-access-k2wdq\") pod \"maas-api-revoke-sub-d3fba056-ckv8s\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.770541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.770454 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ckv8s\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.770541 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.770519 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ckv8s\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.770722 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.770567 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wdq\" (UniqueName: \"kubernetes.io/projected/20ab58dc-1516-416e-b0d8-835301e9f0e2-kube-api-access-k2wdq\") pod \"maas-api-revoke-sub-d3fba056-ckv8s\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.771122 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.771092 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ckv8s\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.772862 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.772840 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ckv8s\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.779437 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.779403 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wdq\" (UniqueName: \"kubernetes.io/projected/20ab58dc-1516-416e-b0d8-835301e9f0e2-kube-api-access-k2wdq\") pod \"maas-api-revoke-sub-d3fba056-ckv8s\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:22.826413 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:22.826392 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:23.102269 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:23.102238 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s"] Nov 01 22:53:23.104164 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:53:23.104136 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ab58dc_1516_416e_b0d8_835301e9f0e2.slice/crio-092ce2da287f677fc0887f34c7a8e672ccff5873aa89e46f6f4fb43be19b2e79 WatchSource:0}: Error finding container 092ce2da287f677fc0887f34c7a8e672ccff5873aa89e46f6f4fb43be19b2e79: Status 404 returned error can't find the container with id 092ce2da287f677fc0887f34c7a8e672ccff5873aa89e46f6f4fb43be19b2e79 Nov 01 22:53:23.881820 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:23.881778 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" event={"ID":"20ab58dc-1516-416e-b0d8-835301e9f0e2","Type":"ContainerStarted","Data":"e4acfe786156c76ffa706858ff5e39709a25d9d4cde8362bdf1333c87a52d767"} Nov 01 22:53:23.882354 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:23.881847 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" event={"ID":"20ab58dc-1516-416e-b0d8-835301e9f0e2","Type":"ContainerStarted","Data":"092ce2da287f677fc0887f34c7a8e672ccff5873aa89e46f6f4fb43be19b2e79"} Nov 01 22:53:23.902997 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:23.902934 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" podStartSLOduration=1.9029161129999999 podStartE2EDuration="1.902916113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:23.899193753 +0000 UTC m=+1816.547692990" watchObservedRunningTime="2026-11-01 22:53:23.902916113 +0000 UTC m=+1816.551415349" Nov 01 22:53:24.961427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:24.961392 6287 generic.go:356] "Generic (PLEG): container finished" podID="20ab58dc-1516-416e-b0d8-835301e9f0e2" containerID="e4acfe786156c76ffa706858ff5e39709a25d9d4cde8362bdf1333c87a52d767" exitCode=0 Nov 01 22:53:24.961943 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:24.961443 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" event={"ID":"20ab58dc-1516-416e-b0d8-835301e9f0e2","Type":"ContainerDied","Data":"e4acfe786156c76ffa706858ff5e39709a25d9d4cde8362bdf1333c87a52d767"} Nov 01 22:53:26.113752 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.113728 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:26.156281 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.156250 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-xks-ca\") pod \"20ab58dc-1516-416e-b0d8-835301e9f0e2\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " Nov 01 22:53:26.156408 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.156330 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-openshift-ca\") pod \"20ab58dc-1516-416e-b0d8-835301e9f0e2\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " Nov 01 22:53:26.156465 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.156426 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ab58dc-1516-416e-b0d8-835301e9f0e2-kube-api-access-k2wdq\" (UniqueName: \"kubernetes.io/projected/20ab58dc-1516-416e-b0d8-835301e9f0e2-kube-api-access-k2wdq\") pod \"20ab58dc-1516-416e-b0d8-835301e9f0e2\" (UID: \"20ab58dc-1516-416e-b0d8-835301e9f0e2\") " Nov 01 22:53:26.156666 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.156641 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-openshift-ca" pod "20ab58dc-1516-416e-b0d8-835301e9f0e2" (UID: "20ab58dc-1516-416e-b0d8-835301e9f0e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:53:26.156784 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.156762 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:26.158310 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.158285 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-xks-ca" pod "20ab58dc-1516-416e-b0d8-835301e9f0e2" (UID: "20ab58dc-1516-416e-b0d8-835301e9f0e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:53:26.158424 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.158401 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ab58dc-1516-416e-b0d8-835301e9f0e2-kube-api-access-k2wdq" pod "20ab58dc-1516-416e-b0d8-835301e9f0e2" (UID: "20ab58dc-1516-416e-b0d8-835301e9f0e2"). InnerVolumeSpecName "kube-api-access-k2wdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:53:26.257726 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.257694 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2wdq\" (UniqueName: \"kubernetes.io/projected/20ab58dc-1516-416e-b0d8-835301e9f0e2-kube-api-access-k2wdq\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:26.257726 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.257724 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ab58dc-1516-416e-b0d8-835301e9f0e2-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:26.974596 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.974564 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" Nov 01 22:53:26.974789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.974601 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s" event={"ID":"20ab58dc-1516-416e-b0d8-835301e9f0e2","Type":"ContainerDied","Data":"092ce2da287f677fc0887f34c7a8e672ccff5873aa89e46f6f4fb43be19b2e79"} Nov 01 22:53:26.974789 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:26.974653 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="092ce2da287f677fc0887f34c7a8e672ccff5873aa89e46f6f4fb43be19b2e79" Nov 01 22:53:31.765575 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.765546 6287 scope.go:122] "RemoveContainer" containerID="ddf066581c064429aae0f545e13e7d851cdec53f855a6ffbdec8eeff54d76fed" Nov 01 22:53:31.774765 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.774743 6287 scope.go:122] "RemoveContainer" containerID="293071a92c5c5d68baa904d7dc1e58b09049ffa09ec2ea6485a377e2eae5ee47" Nov 01 22:53:31.782953 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.782932 6287 scope.go:122] "RemoveContainer" containerID="ad69bb190822825c349f11f3f28840b3d3c4b0cb06f336458bb8d00f03c25716" Nov 01 22:53:31.799754 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.799729 6287 scope.go:122] "RemoveContainer" containerID="129a8c1263f4b9b6794d086cbfcd92d88e46c83900a049588cf7d92d55317d0c" Nov 01 22:53:31.807786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.807761 6287 scope.go:122] "RemoveContainer" containerID="d4bde02e4c4203ca208ef6cb5b41b532e92e3e25f057a1a913015b18a99633ce" Nov 01 22:53:31.815795 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.815775 6287 scope.go:122] "RemoveContainer" containerID="db0b73dfd82e7fe76ee742f6314e14f5c6e56119f1132e700100a6a505138886" Nov 01 22:53:31.823617 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.823598 6287 scope.go:122] "RemoveContainer" containerID="4945f548f5e5340df87933c192e1498a2de7b5ecda847e92d2dd348194db5ace" Nov 01 22:53:31.831421 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.831400 6287 scope.go:122] "RemoveContainer" containerID="d51e704c7ef3fa82479bd51e90a504451e29bb80ca66b2bd66ad7136d19314c2" Nov 01 22:53:31.839093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.839071 6287 scope.go:122] "RemoveContainer" containerID="e14bd6e5ae8e6694600d8d2bafa4ca0af247bdafebbe2ae3c829e2cd56a97fc0" Nov 01 22:53:31.846599 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:31.846572 6287 scope.go:122] "RemoveContainer" containerID="4f927880eb79c6ae35e50475575dd604a3fde5ec59a67585c36b55d94a5083d0" Nov 01 22:53:43.059183 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.059141 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f"] Nov 01 22:53:43.059691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.059672 6287 cpu_manager.go:414] "containerMap: removing container" podUID="20ab58dc-1516-416e-b0d8-835301e9f0e2" containerName="revoke-keys" Nov 01 22:53:43.059766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.059693 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ab58dc-1516-416e-b0d8-835301e9f0e2" containerName="revoke-keys" Nov 01 22:53:43.059857 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.059842 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ab58dc-1516-416e-b0d8-835301e9f0e2" containerName="revoke-keys" Nov 01 22:53:43.065031 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.065007 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.070906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.070877 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:53:43.071065 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.070906 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:53:43.071319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.071289 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:53:43.071491 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.071458 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:53:43.073769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.073747 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f"] Nov 01 22:53:43.130571 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.130535 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-tgg4f\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.130774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.130599 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-tgg4f\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.130774 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.130724 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncwj\" (UniqueName: \"kubernetes.io/projected/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-kube-api-access-gncwj\") pod \"maas-api-revoke-sub-ae594979-tgg4f\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.237013 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.236970 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-tgg4f\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.237236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.237048 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncwj\" (UniqueName: \"kubernetes.io/projected/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-kube-api-access-gncwj\") pod \"maas-api-revoke-sub-ae594979-tgg4f\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.237236 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.237157 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-tgg4f\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.238005 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.237974 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-tgg4f\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.240741 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.240717 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-tgg4f\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.247701 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.247674 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncwj\" (UniqueName: \"kubernetes.io/projected/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-kube-api-access-gncwj\") pod \"maas-api-revoke-sub-ae594979-tgg4f\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.381751 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.381657 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:43.667156 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:43.667062 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f"] Nov 01 22:53:43.670357 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:53:43.670323 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2e682c_14f8_4e86_aa6e_8b7d6b97d32b.slice/crio-48455e6711fc69531c76d699e499aa9e874bc58b270a033fd56ef4b3b2c631cb WatchSource:0}: Error finding container 48455e6711fc69531c76d699e499aa9e874bc58b270a033fd56ef4b3b2c631cb: Status 404 returned error can't find the container with id 48455e6711fc69531c76d699e499aa9e874bc58b270a033fd56ef4b3b2c631cb Nov 01 22:53:44.024440 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:44.024410 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x"] Nov 01 22:53:44.026737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:44.026714 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9d2bd0c-2ml8x"] Nov 01 22:53:44.093225 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:44.093186 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" event={"ID":"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b","Type":"ContainerStarted","Data":"9681cfd6ac875b41c226b9c632c6d1e05c1635eebc256d7ed33d4213e680a264"} Nov 01 22:53:44.093582 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:44.093242 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" event={"ID":"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b","Type":"ContainerStarted","Data":"48455e6711fc69531c76d699e499aa9e874bc58b270a033fd56ef4b3b2c631cb"} Nov 01 22:53:44.110944 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:44.110904 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" podStartSLOduration=1.110891912 podStartE2EDuration="1.110891912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:44.109343125 +0000 UTC m=+1836.757842361" watchObservedRunningTime="2026-11-01 22:53:44.110891912 +0000 UTC m=+1836.759391147" Nov 01 22:53:45.099669 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:45.099633 6287 generic.go:356] "Generic (PLEG): container finished" podID="ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b" containerID="9681cfd6ac875b41c226b9c632c6d1e05c1635eebc256d7ed33d4213e680a264" exitCode=0 Nov 01 22:53:45.100107 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:45.099689 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" event={"ID":"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b","Type":"ContainerDied","Data":"9681cfd6ac875b41c226b9c632c6d1e05c1635eebc256d7ed33d4213e680a264"} Nov 01 22:53:45.473142 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:45.473051 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45bff268-bba1-4559-a6d8-937c1480de8a" path="/var/lib/kubelet/pods/45bff268-bba1-4559-a6d8-937c1480de8a/volumes" Nov 01 22:53:45.587276 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:53:45.587238 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:53:46.257837 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.257809 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:46.364758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.364674 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-xks-ca\") pod \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " Nov 01 22:53:46.364758 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.364716 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-openshift-ca\") pod \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " Nov 01 22:53:46.364967 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.364913 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-kube-api-access-gncwj\" (UniqueName: \"kubernetes.io/projected/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-kube-api-access-gncwj\") pod \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\" (UID: \"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b\") " Nov 01 22:53:46.365135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.365087 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-openshift-ca" pod "ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b" (UID: "ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:53:46.365328 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.365311 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:46.366906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.366872 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-xks-ca" pod "ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b" (UID: "ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:53:46.367022 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.366984 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-kube-api-access-gncwj" pod "ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b" (UID: "ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b"). InnerVolumeSpecName "kube-api-access-gncwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:53:46.466213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.466181 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:46.466213 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:46.466208 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gncwj\" (UniqueName: \"kubernetes.io/projected/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b-kube-api-access-gncwj\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:47.119809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:47.115432 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" event={"ID":"ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b","Type":"ContainerDied","Data":"48455e6711fc69531c76d699e499aa9e874bc58b270a033fd56ef4b3b2c631cb"} Nov 01 22:53:47.119809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:47.115564 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48455e6711fc69531c76d699e499aa9e874bc58b270a033fd56ef4b3b2c631cb" Nov 01 22:53:47.119809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:47.115663 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f" Nov 01 22:53:55.021396 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:55.021352 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p"] Nov 01 22:53:55.023359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:55.023329 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0f8c629-c6d6p"] Nov 01 22:53:55.469207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:53:55.469108 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b3cb34-8631-4aeb-b707-ba93f95dfb82" path="/var/lib/kubelet/pods/75b3cb34-8631-4aeb-b707-ba93f95dfb82/volumes" Nov 01 22:54:01.164295 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.164261 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n"] Nov 01 22:54:01.164684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.164647 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b" containerName="revoke-keys" Nov 01 22:54:01.164684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.164658 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b" containerName="revoke-keys" Nov 01 22:54:01.164764 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.164725 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b" containerName="revoke-keys" Nov 01 22:54:01.167753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.167735 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.172326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.172303 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:54:01.172326 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.172325 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:54:01.172481 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.172324 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:54:01.172481 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.172355 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:54:01.176739 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.176715 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n"] Nov 01 22:54:01.325970 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.325941 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fs77n\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.326188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.325974 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpwr\" (UniqueName: \"kubernetes.io/projected/7cf87316-b82b-45cb-98fe-7cf529091397-kube-api-access-wvpwr\") pod \"maas-api-revoke-sub-a4aef7d7-fs77n\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.326188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.326148 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fs77n\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.426977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.426901 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fs77n\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.426977 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.426939 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpwr\" (UniqueName: \"kubernetes.io/projected/7cf87316-b82b-45cb-98fe-7cf529091397-kube-api-access-wvpwr\") pod \"maas-api-revoke-sub-a4aef7d7-fs77n\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.427179 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.427011 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fs77n\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.427698 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.427666 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fs77n\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.429286 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.429263 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-fs77n\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.435438 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.435415 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpwr\" (UniqueName: \"kubernetes.io/projected/7cf87316-b82b-45cb-98fe-7cf529091397-kube-api-access-wvpwr\") pod \"maas-api-revoke-sub-a4aef7d7-fs77n\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.482077 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.482052 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:01.762253 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:01.762222 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n"] Nov 01 22:54:01.762849 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:54:01.762818 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf87316_b82b_45cb_98fe_7cf529091397.slice/crio-2233fd3d1850441b68eb7a32227a95ae0fcdb0da0dad47ad71fdc45dbe010509 WatchSource:0}: Error finding container 2233fd3d1850441b68eb7a32227a95ae0fcdb0da0dad47ad71fdc45dbe010509: Status 404 returned error can't find the container with id 2233fd3d1850441b68eb7a32227a95ae0fcdb0da0dad47ad71fdc45dbe010509 Nov 01 22:54:02.220568 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:02.220532 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" event={"ID":"7cf87316-b82b-45cb-98fe-7cf529091397","Type":"ContainerStarted","Data":"78aa489990c3b2ff5d994e5fc298027cf5e7418dcc0a62c04d1bbf2260637576"} Nov 01 22:54:02.220921 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:02.220598 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" event={"ID":"7cf87316-b82b-45cb-98fe-7cf529091397","Type":"ContainerStarted","Data":"2233fd3d1850441b68eb7a32227a95ae0fcdb0da0dad47ad71fdc45dbe010509"} Nov 01 22:54:02.241002 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:02.240958 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" podStartSLOduration=1.240945231 podStartE2EDuration="1.240945231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:02.239362906 +0000 UTC m=+1854.887862143" watchObservedRunningTime="2026-11-01 22:54:02.240945231 +0000 UTC m=+1854.889444444" Nov 01 22:54:03.227573 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:03.227538 6287 generic.go:356] "Generic (PLEG): container finished" podID="7cf87316-b82b-45cb-98fe-7cf529091397" containerID="78aa489990c3b2ff5d994e5fc298027cf5e7418dcc0a62c04d1bbf2260637576" exitCode=0 Nov 01 22:54:03.228105 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:03.227616 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" event={"ID":"7cf87316-b82b-45cb-98fe-7cf529091397","Type":"ContainerDied","Data":"78aa489990c3b2ff5d994e5fc298027cf5e7418dcc0a62c04d1bbf2260637576"} Nov 01 22:54:04.365540 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.365514 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:04.454589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.454560 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-openshift-ca\") pod \"7cf87316-b82b-45cb-98fe-7cf529091397\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " Nov 01 22:54:04.454734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.454609 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf87316-b82b-45cb-98fe-7cf529091397-kube-api-access-wvpwr\" (UniqueName: \"kubernetes.io/projected/7cf87316-b82b-45cb-98fe-7cf529091397-kube-api-access-wvpwr\") pod \"7cf87316-b82b-45cb-98fe-7cf529091397\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " Nov 01 22:54:04.454734 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.454703 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-xks-ca\") pod \"7cf87316-b82b-45cb-98fe-7cf529091397\" (UID: \"7cf87316-b82b-45cb-98fe-7cf529091397\") " Nov 01 22:54:04.454911 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.454890 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-openshift-ca" pod "7cf87316-b82b-45cb-98fe-7cf529091397" (UID: "7cf87316-b82b-45cb-98fe-7cf529091397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:04.456766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.456736 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf87316-b82b-45cb-98fe-7cf529091397-kube-api-access-wvpwr" pod "7cf87316-b82b-45cb-98fe-7cf529091397" (UID: "7cf87316-b82b-45cb-98fe-7cf529091397"). InnerVolumeSpecName "kube-api-access-wvpwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:04.456868 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.456796 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-xks-ca" pod "7cf87316-b82b-45cb-98fe-7cf529091397" (UID: "7cf87316-b82b-45cb-98fe-7cf529091397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:04.555423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.555392 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvpwr\" (UniqueName: \"kubernetes.io/projected/7cf87316-b82b-45cb-98fe-7cf529091397-kube-api-access-wvpwr\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:04.555423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.555419 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:04.555423 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:04.555430 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf87316-b82b-45cb-98fe-7cf529091397-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:05.239504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:05.239468 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" Nov 01 22:54:05.239504 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:05.239496 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n" event={"ID":"7cf87316-b82b-45cb-98fe-7cf529091397","Type":"ContainerDied","Data":"2233fd3d1850441b68eb7a32227a95ae0fcdb0da0dad47ad71fdc45dbe010509"} Nov 01 22:54:05.239737 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:05.239545 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2233fd3d1850441b68eb7a32227a95ae0fcdb0da0dad47ad71fdc45dbe010509" Nov 01 22:54:15.650086 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:54:15.650009 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:54:19.904285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.904250 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7"] Nov 01 22:54:19.904762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.904624 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7cf87316-b82b-45cb-98fe-7cf529091397" containerName="revoke-keys" Nov 01 22:54:19.904762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.904636 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf87316-b82b-45cb-98fe-7cf529091397" containerName="revoke-keys" Nov 01 22:54:19.904762 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.904757 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf87316-b82b-45cb-98fe-7cf529091397" containerName="revoke-keys" Nov 01 22:54:19.908085 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.908063 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:19.916319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.916052 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:54:19.916319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.916225 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:54:19.916319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.916273 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:54:19.916319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.916276 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:54:19.923637 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.923601 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7"] Nov 01 22:54:19.994610 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.994578 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hf9n7\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:19.994812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.994631 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqjd\" (UniqueName: \"kubernetes.io/projected/85f5bc2f-df8d-4fee-8b97-33dd89874449-kube-api-access-xsqjd\") pod \"maas-api-revoke-sub-79805ca9-hf9n7\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:19.994812 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:19.994765 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hf9n7\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:20.095711 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:20.095669 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hf9n7\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:20.095867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:20.095749 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hf9n7\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:20.095867 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:20.095806 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqjd\" (UniqueName: \"kubernetes.io/projected/85f5bc2f-df8d-4fee-8b97-33dd89874449-kube-api-access-xsqjd\") pod \"maas-api-revoke-sub-79805ca9-hf9n7\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:20.096419 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:20.096398 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-hf9n7\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:20.098027 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:20.097996 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-hf9n7\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:20.104221 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:20.104200 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqjd\" (UniqueName: \"kubernetes.io/projected/85f5bc2f-df8d-4fee-8b97-33dd89874449-kube-api-access-xsqjd\") pod \"maas-api-revoke-sub-79805ca9-hf9n7\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:20.233083 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:20.233022 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:20.505769 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:20.505742 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7"] Nov 01 22:54:20.507675 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:54:20.507649 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f5bc2f_df8d_4fee_8b97_33dd89874449.slice/crio-de47d2abe4094b0d273ffdf6076a9a63ba73d768c1951164c81a5483d3f080cd WatchSource:0}: Error finding container de47d2abe4094b0d273ffdf6076a9a63ba73d768c1951164c81a5483d3f080cd: Status 404 returned error can't find the container with id de47d2abe4094b0d273ffdf6076a9a63ba73d768c1951164c81a5483d3f080cd Nov 01 22:54:21.378435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:21.378403 6287 generic.go:356] "Generic (PLEG): container finished" podID="85f5bc2f-df8d-4fee-8b97-33dd89874449" containerID="289ef4201cce2b004b8b2c0f03b9192bf70d1522d5b0b9cc523deca1845f9861" exitCode=0 Nov 01 22:54:21.378785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:21.378492 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" event={"ID":"85f5bc2f-df8d-4fee-8b97-33dd89874449","Type":"ContainerDied","Data":"289ef4201cce2b004b8b2c0f03b9192bf70d1522d5b0b9cc523deca1845f9861"} Nov 01 22:54:21.378785 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:21.378546 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" event={"ID":"85f5bc2f-df8d-4fee-8b97-33dd89874449","Type":"ContainerStarted","Data":"de47d2abe4094b0d273ffdf6076a9a63ba73d768c1951164c81a5483d3f080cd"} Nov 01 22:54:22.518549 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.518521 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:22.615622 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.615588 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-openshift-ca\") pod \"85f5bc2f-df8d-4fee-8b97-33dd89874449\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " Nov 01 22:54:22.615790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.615672 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-xks-ca\") pod \"85f5bc2f-df8d-4fee-8b97-33dd89874449\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " Nov 01 22:54:22.615790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.615703 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f5bc2f-df8d-4fee-8b97-33dd89874449-kube-api-access-xsqjd\" (UniqueName: \"kubernetes.io/projected/85f5bc2f-df8d-4fee-8b97-33dd89874449-kube-api-access-xsqjd\") pod \"85f5bc2f-df8d-4fee-8b97-33dd89874449\" (UID: \"85f5bc2f-df8d-4fee-8b97-33dd89874449\") " Nov 01 22:54:22.615957 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.615934 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-openshift-ca" pod "85f5bc2f-df8d-4fee-8b97-33dd89874449" (UID: "85f5bc2f-df8d-4fee-8b97-33dd89874449"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:22.617895 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.617870 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f5bc2f-df8d-4fee-8b97-33dd89874449-kube-api-access-xsqjd" pod "85f5bc2f-df8d-4fee-8b97-33dd89874449" (UID: "85f5bc2f-df8d-4fee-8b97-33dd89874449"). InnerVolumeSpecName "kube-api-access-xsqjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:22.618019 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.617916 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-xks-ca" pod "85f5bc2f-df8d-4fee-8b97-33dd89874449" (UID: "85f5bc2f-df8d-4fee-8b97-33dd89874449"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:22.717228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.717142 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:22.717228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.717173 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f5bc2f-df8d-4fee-8b97-33dd89874449-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:22.717228 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:22.717185 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqjd\" (UniqueName: \"kubernetes.io/projected/85f5bc2f-df8d-4fee-8b97-33dd89874449-kube-api-access-xsqjd\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:23.390021 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:23.389986 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" Nov 01 22:54:23.390252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:23.390023 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7" event={"ID":"85f5bc2f-df8d-4fee-8b97-33dd89874449","Type":"ContainerDied","Data":"de47d2abe4094b0d273ffdf6076a9a63ba73d768c1951164c81a5483d3f080cd"} Nov 01 22:54:23.390252 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:23.390072 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de47d2abe4094b0d273ffdf6076a9a63ba73d768c1951164c81a5483d3f080cd" Nov 01 22:54:37.587566 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.587530 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b"] Nov 01 22:54:37.588054 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.588039 6287 cpu_manager.go:414] "containerMap: removing container" podUID="85f5bc2f-df8d-4fee-8b97-33dd89874449" containerName="revoke-keys" Nov 01 22:54:37.588106 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.588056 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f5bc2f-df8d-4fee-8b97-33dd89874449" containerName="revoke-keys" Nov 01 22:54:37.588188 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.588176 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f5bc2f-df8d-4fee-8b97-33dd89874449" containerName="revoke-keys" Nov 01 22:54:37.591817 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.591796 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.599082 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.599056 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:54:37.599246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.599162 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:54:37.599246 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.599197 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:54:37.599435 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.599417 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:54:37.600753 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.600726 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b"] Nov 01 22:54:37.669224 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.669192 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cbz2b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.669403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.669236 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vztst\" (UniqueName: \"kubernetes.io/projected/de1b74ef-2cd1-47c2-b619-c6661dbe024b-kube-api-access-vztst\") pod \"maas-api-revoke-sub-0b7e8edb-cbz2b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.669403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.669353 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cbz2b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.770715 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.770676 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cbz2b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.770920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.770728 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vztst\" (UniqueName: \"kubernetes.io/projected/de1b74ef-2cd1-47c2-b619-c6661dbe024b-kube-api-access-vztst\") pod \"maas-api-revoke-sub-0b7e8edb-cbz2b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.770920 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.770793 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cbz2b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.771339 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.771316 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cbz2b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.773078 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.773062 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-cbz2b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.779238 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.779219 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vztst\" (UniqueName: \"kubernetes.io/projected/de1b74ef-2cd1-47c2-b619-c6661dbe024b-kube-api-access-vztst\") pod \"maas-api-revoke-sub-0b7e8edb-cbz2b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:37.910025 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:37.909938 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:38.183223 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:38.183195 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b"] Nov 01 22:54:38.185326 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:54:38.185290 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1b74ef_2cd1_47c2_b619_c6661dbe024b.slice/crio-251ff9e12c30887cab90ac0db7c00c9c9723166b5896a5e1b94fa1187407cfba WatchSource:0}: Error finding container 251ff9e12c30887cab90ac0db7c00c9c9723166b5896a5e1b94fa1187407cfba: Status 404 returned error can't find the container with id 251ff9e12c30887cab90ac0db7c00c9c9723166b5896a5e1b94fa1187407cfba Nov 01 22:54:38.479442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:38.479359 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" event={"ID":"de1b74ef-2cd1-47c2-b619-c6661dbe024b","Type":"ContainerStarted","Data":"cbd7ce6df76657fa723bac14890eae7a9cff8e21ab11d12a9a61ffb713767a1e"} Nov 01 22:54:38.479442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:38.479421 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" event={"ID":"de1b74ef-2cd1-47c2-b619-c6661dbe024b","Type":"ContainerStarted","Data":"251ff9e12c30887cab90ac0db7c00c9c9723166b5896a5e1b94fa1187407cfba"} Nov 01 22:54:38.497589 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:38.497536 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" podStartSLOduration=1.4975239949999999 podStartE2EDuration="1.497523995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:38.496189382 +0000 UTC m=+1891.144688623" watchObservedRunningTime="2026-11-01 22:54:38.497523995 +0000 UTC m=+1891.146023230" Nov 01 22:54:39.486016 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:39.485979 6287 generic.go:356] "Generic (PLEG): container finished" podID="de1b74ef-2cd1-47c2-b619-c6661dbe024b" containerID="cbd7ce6df76657fa723bac14890eae7a9cff8e21ab11d12a9a61ffb713767a1e" exitCode=0 Nov 01 22:54:39.486443 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:39.486057 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" event={"ID":"de1b74ef-2cd1-47c2-b619-c6661dbe024b","Type":"ContainerDied","Data":"cbd7ce6df76657fa723bac14890eae7a9cff8e21ab11d12a9a61ffb713767a1e"} Nov 01 22:54:40.623766 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.623740 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:40.696192 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.696157 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1b74ef-2cd1-47c2-b619-c6661dbe024b-kube-api-access-vztst\" (UniqueName: \"kubernetes.io/projected/de1b74ef-2cd1-47c2-b619-c6661dbe024b-kube-api-access-vztst\") pod \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " Nov 01 22:54:40.696399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.696216 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-xks-ca\") pod \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " Nov 01 22:54:40.696399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.696265 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-openshift-ca\") pod \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\" (UID: \"de1b74ef-2cd1-47c2-b619-c6661dbe024b\") " Nov 01 22:54:40.696699 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.696672 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-openshift-ca" pod "de1b74ef-2cd1-47c2-b619-c6661dbe024b" (UID: "de1b74ef-2cd1-47c2-b619-c6661dbe024b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:40.698403 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.698385 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-xks-ca" pod "de1b74ef-2cd1-47c2-b619-c6661dbe024b" (UID: "de1b74ef-2cd1-47c2-b619-c6661dbe024b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:40.698553 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.698532 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1b74ef-2cd1-47c2-b619-c6661dbe024b-kube-api-access-vztst" pod "de1b74ef-2cd1-47c2-b619-c6661dbe024b" (UID: "de1b74ef-2cd1-47c2-b619-c6661dbe024b"). InnerVolumeSpecName "kube-api-access-vztst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:40.797654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.797570 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vztst\" (UniqueName: \"kubernetes.io/projected/de1b74ef-2cd1-47c2-b619-c6661dbe024b-kube-api-access-vztst\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:40.797654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.797603 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:40.797654 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:40.797613 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1b74ef-2cd1-47c2-b619-c6661dbe024b-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:41.497430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:41.497404 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" Nov 01 22:54:41.497430 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:41.497410 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b" event={"ID":"de1b74ef-2cd1-47c2-b619-c6661dbe024b","Type":"ContainerDied","Data":"251ff9e12c30887cab90ac0db7c00c9c9723166b5896a5e1b94fa1187407cfba"} Nov 01 22:54:41.497626 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:41.497466 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="251ff9e12c30887cab90ac0db7c00c9c9723166b5896a5e1b94fa1187407cfba" Nov 01 22:54:45.595011 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:54:45.594969 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:54:48.242554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.242478 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs"] Nov 01 22:54:48.242965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.242844 6287 cpu_manager.go:414] "containerMap: removing container" podUID="de1b74ef-2cd1-47c2-b619-c6661dbe024b" containerName="revoke-keys" Nov 01 22:54:48.242965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.242855 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1b74ef-2cd1-47c2-b619-c6661dbe024b" containerName="revoke-keys" Nov 01 22:54:48.242965 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.242943 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1b74ef-2cd1-47c2-b619-c6661dbe024b" containerName="revoke-keys" Nov 01 22:54:48.246153 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.246131 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.255071 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.255033 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs"] Nov 01 22:54:48.256149 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.256104 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:54:48.256274 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.256194 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:54:48.256521 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.256501 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:54:48.256667 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.256650 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:54:48.363519 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.363487 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c5shs\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.363685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.363607 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c5shs\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.363685 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.363636 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4r44\" (UniqueName: \"kubernetes.io/projected/83440853-5b49-427c-ae8b-f8d3241d9f2f-kube-api-access-c4r44\") pod \"maas-api-revoke-sub-3d4156f1-c5shs\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.464186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.464156 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c5shs\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.464356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.464235 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c5shs\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.464356 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.464275 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4r44\" (UniqueName: \"kubernetes.io/projected/83440853-5b49-427c-ae8b-f8d3241d9f2f-kube-api-access-c4r44\") pod \"maas-api-revoke-sub-3d4156f1-c5shs\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.464905 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.464882 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c5shs\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.466459 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.466439 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c5shs\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.473900 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.473880 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4r44\" (UniqueName: \"kubernetes.io/projected/83440853-5b49-427c-ae8b-f8d3241d9f2f-kube-api-access-c4r44\") pod \"maas-api-revoke-sub-3d4156f1-c5shs\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.563641 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.563567 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:48.836744 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:48.836718 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs"] Nov 01 22:54:48.838920 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:54:48.838891 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83440853_5b49_427c_ae8b_f8d3241d9f2f.slice/crio-259a017a84325e7a7043eac223816f4cbfaed2d4d1548a7f7e5ca8104889d32a WatchSource:0}: Error finding container 259a017a84325e7a7043eac223816f4cbfaed2d4d1548a7f7e5ca8104889d32a: Status 404 returned error can't find the container with id 259a017a84325e7a7043eac223816f4cbfaed2d4d1548a7f7e5ca8104889d32a Nov 01 22:54:49.540853 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:49.540813 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" event={"ID":"83440853-5b49-427c-ae8b-f8d3241d9f2f","Type":"ContainerStarted","Data":"1317388fe5d61480a0e0585eca4e3e90894880dbc12ef43cdcab220ad0e4d278"} Nov 01 22:54:49.541267 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:49.540883 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" event={"ID":"83440853-5b49-427c-ae8b-f8d3241d9f2f","Type":"ContainerStarted","Data":"259a017a84325e7a7043eac223816f4cbfaed2d4d1548a7f7e5ca8104889d32a"} Nov 01 22:54:49.558135 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:49.558072 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" podStartSLOduration=1.558058511 podStartE2EDuration="1.558058511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:49.55702997 +0000 UTC m=+1902.205529204" watchObservedRunningTime="2026-11-01 22:54:49.558058511 +0000 UTC m=+1902.206557745" Nov 01 22:54:50.547842 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:50.547806 6287 generic.go:356] "Generic (PLEG): container finished" podID="83440853-5b49-427c-ae8b-f8d3241d9f2f" containerID="1317388fe5d61480a0e0585eca4e3e90894880dbc12ef43cdcab220ad0e4d278" exitCode=0 Nov 01 22:54:50.548254 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:50.547898 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" event={"ID":"83440853-5b49-427c-ae8b-f8d3241d9f2f","Type":"ContainerDied","Data":"1317388fe5d61480a0e0585eca4e3e90894880dbc12ef43cdcab220ad0e4d278"} Nov 01 22:54:51.683988 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.683966 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:51.794088 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.794040 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83440853-5b49-427c-ae8b-f8d3241d9f2f-kube-api-access-c4r44\" (UniqueName: \"kubernetes.io/projected/83440853-5b49-427c-ae8b-f8d3241d9f2f-kube-api-access-c4r44\") pod \"83440853-5b49-427c-ae8b-f8d3241d9f2f\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " Nov 01 22:54:51.794339 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.794137 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-openshift-ca\") pod \"83440853-5b49-427c-ae8b-f8d3241d9f2f\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " Nov 01 22:54:51.794339 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.794173 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-xks-ca\") pod \"83440853-5b49-427c-ae8b-f8d3241d9f2f\" (UID: \"83440853-5b49-427c-ae8b-f8d3241d9f2f\") " Nov 01 22:54:51.794494 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.794471 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-openshift-ca" pod "83440853-5b49-427c-ae8b-f8d3241d9f2f" (UID: "83440853-5b49-427c-ae8b-f8d3241d9f2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:51.796359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.796328 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-xks-ca" pod "83440853-5b49-427c-ae8b-f8d3241d9f2f" (UID: "83440853-5b49-427c-ae8b-f8d3241d9f2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:51.796481 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.796388 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83440853-5b49-427c-ae8b-f8d3241d9f2f-kube-api-access-c4r44" pod "83440853-5b49-427c-ae8b-f8d3241d9f2f" (UID: "83440853-5b49-427c-ae8b-f8d3241d9f2f"). InnerVolumeSpecName "kube-api-access-c4r44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:51.895692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.895607 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4r44\" (UniqueName: \"kubernetes.io/projected/83440853-5b49-427c-ae8b-f8d3241d9f2f-kube-api-access-c4r44\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:51.895692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.895638 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:51.895692 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:51.895649 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83440853-5b49-427c-ae8b-f8d3241d9f2f-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:52.559988 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:52.559953 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" Nov 01 22:54:52.560222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:52.559985 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs" event={"ID":"83440853-5b49-427c-ae8b-f8d3241d9f2f","Type":"ContainerDied","Data":"259a017a84325e7a7043eac223816f4cbfaed2d4d1548a7f7e5ca8104889d32a"} Nov 01 22:54:52.560222 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:54:52.560046 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259a017a84325e7a7043eac223816f4cbfaed2d4d1548a7f7e5ca8104889d32a" Nov 01 22:55:06.175886 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.175847 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs"] Nov 01 22:55:06.177771 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.177747 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c5shs"] Nov 01 22:55:06.199093 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.199050 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j"] Nov 01 22:55:06.199427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.199409 6287 cpu_manager.go:414] "containerMap: removing container" podUID="83440853-5b49-427c-ae8b-f8d3241d9f2f" containerName="revoke-keys" Nov 01 22:55:06.199427 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.199423 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83440853-5b49-427c-ae8b-f8d3241d9f2f" containerName="revoke-keys" Nov 01 22:55:06.199639 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.199507 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="83440853-5b49-427c-ae8b-f8d3241d9f2f" containerName="revoke-keys" Nov 01 22:55:06.202658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.202634 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.206906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.206876 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:55:06.206906 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.206876 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:55:06.207099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.206940 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:55:06.207099 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.207009 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:55:06.213249 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.212753 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j"] Nov 01 22:55:06.266484 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.266450 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gr8j\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.266651 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.266499 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gr8j\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.266697 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.266646 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p29dm\" (UniqueName: \"kubernetes.io/projected/a34a7fe8-0b0a-4da6-b6a6-43b564852536-kube-api-access-p29dm\") pod \"maas-api-revoke-sub-3d4156f1-6gr8j\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.367415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.367369 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gr8j\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.367415 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.367422 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gr8j\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.367623 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.367479 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p29dm\" (UniqueName: \"kubernetes.io/projected/a34a7fe8-0b0a-4da6-b6a6-43b564852536-kube-api-access-p29dm\") pod \"maas-api-revoke-sub-3d4156f1-6gr8j\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.368205 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.368184 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gr8j\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.369675 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.369655 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gr8j\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.376797 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.376773 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p29dm\" (UniqueName: \"kubernetes.io/projected/a34a7fe8-0b0a-4da6-b6a6-43b564852536-kube-api-access-p29dm\") pod \"maas-api-revoke-sub-3d4156f1-6gr8j\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.516368 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.516342 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:06.788323 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:06.788294 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j"] Nov 01 22:55:06.790071 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:55:06.790047 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34a7fe8_0b0a_4da6_b6a6_43b564852536.slice/crio-b9886b11eaad73f128cd4a842eb5d6c7ddd4fadc4ba2b4e1f6609c96a9448768 WatchSource:0}: Error finding container b9886b11eaad73f128cd4a842eb5d6c7ddd4fadc4ba2b4e1f6609c96a9448768: Status 404 returned error can't find the container with id b9886b11eaad73f128cd4a842eb5d6c7ddd4fadc4ba2b4e1f6609c96a9448768 Nov 01 22:55:07.469207 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:07.469133 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83440853-5b49-427c-ae8b-f8d3241d9f2f" path="/var/lib/kubelet/pods/83440853-5b49-427c-ae8b-f8d3241d9f2f/volumes" Nov 01 22:55:07.681042 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:07.681009 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" event={"ID":"a34a7fe8-0b0a-4da6-b6a6-43b564852536","Type":"ContainerStarted","Data":"17a73c8053f3410f5fba4872ff45d7fc0cd531b97eeca4964705a2ba93234387"} Nov 01 22:55:07.681197 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:07.681068 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" event={"ID":"a34a7fe8-0b0a-4da6-b6a6-43b564852536","Type":"ContainerStarted","Data":"b9886b11eaad73f128cd4a842eb5d6c7ddd4fadc4ba2b4e1f6609c96a9448768"} Nov 01 22:55:08.688446 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.688415 6287 generic.go:356] "Generic (PLEG): container finished" podID="a34a7fe8-0b0a-4da6-b6a6-43b564852536" containerID="17a73c8053f3410f5fba4872ff45d7fc0cd531b97eeca4964705a2ba93234387" exitCode=0 Nov 01 22:55:08.688863 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.688498 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" event={"ID":"a34a7fe8-0b0a-4da6-b6a6-43b564852536","Type":"ContainerDied","Data":"17a73c8053f3410f5fba4872ff45d7fc0cd531b97eeca4964705a2ba93234387"} Nov 01 22:55:08.828244 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.828219 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:08.889830 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.889801 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-xks-ca\") pod \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " Nov 01 22:55:08.890010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.889895 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-openshift-ca\") pod \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " Nov 01 22:55:08.890010 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.889922 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34a7fe8-0b0a-4da6-b6a6-43b564852536-kube-api-access-p29dm\" (UniqueName: \"kubernetes.io/projected/a34a7fe8-0b0a-4da6-b6a6-43b564852536-kube-api-access-p29dm\") pod \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\" (UID: \"a34a7fe8-0b0a-4da6-b6a6-43b564852536\") " Nov 01 22:55:08.890285 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.890258 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-openshift-ca" pod "a34a7fe8-0b0a-4da6-b6a6-43b564852536" (UID: "a34a7fe8-0b0a-4da6-b6a6-43b564852536"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:08.891923 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.891895 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-xks-ca" pod "a34a7fe8-0b0a-4da6-b6a6-43b564852536" (UID: "a34a7fe8-0b0a-4da6-b6a6-43b564852536"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:08.892017 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.891967 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34a7fe8-0b0a-4da6-b6a6-43b564852536-kube-api-access-p29dm" pod "a34a7fe8-0b0a-4da6-b6a6-43b564852536" (UID: "a34a7fe8-0b0a-4da6-b6a6-43b564852536"). InnerVolumeSpecName "kube-api-access-p29dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:08.991332 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.991304 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:08.991442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.991332 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a34a7fe8-0b0a-4da6-b6a6-43b564852536-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:08.991442 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:08.991347 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p29dm\" (UniqueName: \"kubernetes.io/projected/a34a7fe8-0b0a-4da6-b6a6-43b564852536-kube-api-access-p29dm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.695100 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:09.695067 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" Nov 01 22:55:09.695523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:09.695101 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j" event={"ID":"a34a7fe8-0b0a-4da6-b6a6-43b564852536","Type":"ContainerDied","Data":"b9886b11eaad73f128cd4a842eb5d6c7ddd4fadc4ba2b4e1f6609c96a9448768"} Nov 01 22:55:09.695523 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:09.695167 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9886b11eaad73f128cd4a842eb5d6c7ddd4fadc4ba2b4e1f6609c96a9448768" Nov 01 22:55:15.586752 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:55:15.586714 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:55:31.914677 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:31.914645 6287 scope.go:122] "RemoveContainer" containerID="5b768d446583738f5d384b1f61ea1409796373e944f1d08dc86df85afeb1de6a" Nov 01 22:55:31.923630 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:31.923606 6287 scope.go:122] "RemoveContainer" containerID="7f6bded81deb870c90cba829c0c8a826c612dbb028ac5d12c7696be10c15a527" Nov 01 22:55:31.931687 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:31.931669 6287 scope.go:122] "RemoveContainer" containerID="332da1cae60121fcc57f8340e51d83aa26f4c9876197ea1c02cbc5082c355916" Nov 01 22:55:31.939804 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:31.939781 6287 scope.go:122] "RemoveContainer" containerID="819ef50f4aeac62b9342fdbefe4d2e8719d542e590b5888d9dc41658a045f12f" Nov 01 22:55:34.653823 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.653793 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq"] Nov 01 22:55:34.654234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.654172 6287 cpu_manager.go:414] "containerMap: removing container" podUID="a34a7fe8-0b0a-4da6-b6a6-43b564852536" containerName="revoke-keys" Nov 01 22:55:34.654234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.654185 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34a7fe8-0b0a-4da6-b6a6-43b564852536" containerName="revoke-keys" Nov 01 22:55:34.654325 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.654254 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34a7fe8-0b0a-4da6-b6a6-43b564852536" containerName="revoke-keys" Nov 01 22:55:34.658699 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.658681 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.670500 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.670468 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:55:34.671278 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.671254 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:55:34.671414 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.671330 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:55:34.671493 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.671462 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq"] Nov 01 22:55:34.672023 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.672001 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:55:34.691931 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.691895 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p5mgq\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.692070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.691950 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmflt\" (UniqueName: \"kubernetes.io/projected/5dd595b5-f707-4aad-8781-a32ab492a280-kube-api-access-gmflt\") pod \"maas-api-revoke-sub-e2fc9c5b-p5mgq\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.692170 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.692157 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p5mgq\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.793288 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.793253 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p5mgq\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.793471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.793302 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p5mgq\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.793471 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.793326 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmflt\" (UniqueName: \"kubernetes.io/projected/5dd595b5-f707-4aad-8781-a32ab492a280-kube-api-access-gmflt\") pod \"maas-api-revoke-sub-e2fc9c5b-p5mgq\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.793909 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.793881 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p5mgq\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.795691 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.795665 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-p5mgq\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.804339 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.804317 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmflt\" (UniqueName: \"kubernetes.io/projected/5dd595b5-f707-4aad-8781-a32ab492a280-kube-api-access-gmflt\") pod \"maas-api-revoke-sub-e2fc9c5b-p5mgq\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:34.975799 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:34.975722 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:35.247871 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:35.247838 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq"] Nov 01 22:55:35.253221 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:55:35.253187 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd595b5_f707_4aad_8781_a32ab492a280.slice/crio-13e9c2fbf525ba6853fa80843806c3f3895e3a039609790c3d18ce88957d4fe5 WatchSource:0}: Error finding container 13e9c2fbf525ba6853fa80843806c3f3895e3a039609790c3d18ce88957d4fe5: Status 404 returned error can't find the container with id 13e9c2fbf525ba6853fa80843806c3f3895e3a039609790c3d18ce88957d4fe5 Nov 01 22:55:35.874650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:35.874562 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" event={"ID":"5dd595b5-f707-4aad-8781-a32ab492a280","Type":"ContainerStarted","Data":"5d1bb4a7dc95fd67bd13c564cb65db1fe431c7538f2ee632a94638cab2333670"} Nov 01 22:55:35.874650 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:35.874636 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" event={"ID":"5dd595b5-f707-4aad-8781-a32ab492a280","Type":"ContainerStarted","Data":"13e9c2fbf525ba6853fa80843806c3f3895e3a039609790c3d18ce88957d4fe5"} Nov 01 22:55:35.895040 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:35.894990 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" podStartSLOduration=1.894977929 podStartE2EDuration="1.894977929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:35.892759152 +0000 UTC m=+1948.541258387" watchObservedRunningTime="2026-11-01 22:55:35.894977929 +0000 UTC m=+1948.543477163" Nov 01 22:55:36.881359 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:36.881320 6287 generic.go:356] "Generic (PLEG): container finished" podID="5dd595b5-f707-4aad-8781-a32ab492a280" containerID="5d1bb4a7dc95fd67bd13c564cb65db1fe431c7538f2ee632a94638cab2333670" exitCode=0 Nov 01 22:55:36.881773 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:36.881360 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" event={"ID":"5dd595b5-f707-4aad-8781-a32ab492a280","Type":"ContainerDied","Data":"5d1bb4a7dc95fd67bd13c564cb65db1fe431c7538f2ee632a94638cab2333670"} Nov 01 22:55:38.030648 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.030622 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:38.124125 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.124090 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-xks-ca\") pod \"5dd595b5-f707-4aad-8781-a32ab492a280\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " Nov 01 22:55:38.124319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.124206 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd595b5-f707-4aad-8781-a32ab492a280-kube-api-access-gmflt\" (UniqueName: \"kubernetes.io/projected/5dd595b5-f707-4aad-8781-a32ab492a280-kube-api-access-gmflt\") pod \"5dd595b5-f707-4aad-8781-a32ab492a280\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " Nov 01 22:55:38.124319 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.124285 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-openshift-ca\") pod \"5dd595b5-f707-4aad-8781-a32ab492a280\" (UID: \"5dd595b5-f707-4aad-8781-a32ab492a280\") " Nov 01 22:55:38.124603 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.124579 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-openshift-ca" pod "5dd595b5-f707-4aad-8781-a32ab492a280" (UID: "5dd595b5-f707-4aad-8781-a32ab492a280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:38.126210 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.126181 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-xks-ca" pod "5dd595b5-f707-4aad-8781-a32ab492a280" (UID: "5dd595b5-f707-4aad-8781-a32ab492a280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:38.126320 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.126274 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd595b5-f707-4aad-8781-a32ab492a280-kube-api-access-gmflt" pod "5dd595b5-f707-4aad-8781-a32ab492a280" (UID: "5dd595b5-f707-4aad-8781-a32ab492a280"). InnerVolumeSpecName "kube-api-access-gmflt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:38.225658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.225591 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:38.225658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.225616 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmflt\" (UniqueName: \"kubernetes.io/projected/5dd595b5-f707-4aad-8781-a32ab492a280-kube-api-access-gmflt\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:38.225658 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.225627 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd595b5-f707-4aad-8781-a32ab492a280-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:38.893467 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.893432 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" Nov 01 22:55:38.893467 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.893454 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq" event={"ID":"5dd595b5-f707-4aad-8781-a32ab492a280","Type":"ContainerDied","Data":"13e9c2fbf525ba6853fa80843806c3f3895e3a039609790c3d18ce88957d4fe5"} Nov 01 22:55:38.893708 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:38.893506 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13e9c2fbf525ba6853fa80843806c3f3895e3a039609790c3d18ce88957d4fe5" Nov 01 22:55:40.029760 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:40.029726 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd"] Nov 01 22:55:40.037899 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:40.037872 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd96b34-dhkrd"] Nov 01 22:55:41.470595 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:41.470552 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6c080e-a807-48f2-88c7-29ff8bce4cc5" path="/var/lib/kubelet/pods/dd6c080e-a807-48f2-88c7-29ff8bce4cc5/volumes" Nov 01 22:55:44.848870 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.848836 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q"] Nov 01 22:55:44.849426 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.849406 6287 cpu_manager.go:414] "containerMap: removing container" podUID="5dd595b5-f707-4aad-8781-a32ab492a280" containerName="revoke-keys" Nov 01 22:55:44.849526 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.849428 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd595b5-f707-4aad-8781-a32ab492a280" containerName="revoke-keys" Nov 01 22:55:44.849588 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.849528 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd595b5-f707-4aad-8781-a32ab492a280" containerName="revoke-keys" Nov 01 22:55:44.853684 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.853661 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:44.858831 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.858804 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:55:44.858961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.858844 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:55:44.858961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.858887 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 22:55:44.858961 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.858889 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:55:44.861662 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.861641 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q"] Nov 01 22:55:44.978376 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.978341 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5fr9q\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:44.978572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.978391 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9rnv\" (UniqueName: \"kubernetes.io/projected/3a8f86d1-2385-4c9c-92e1-4561c3663f62-kube-api-access-q9rnv\") pod \"maas-api-revoke-sub-5a29efb3-5fr9q\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:44.978572 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:44.978529 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5fr9q\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:45.079725 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.079691 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5fr9q\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:45.079887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.079762 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5fr9q\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:45.079887 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.079799 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9rnv\" (UniqueName: \"kubernetes.io/projected/3a8f86d1-2385-4c9c-92e1-4561c3663f62-kube-api-access-q9rnv\") pod \"maas-api-revoke-sub-5a29efb3-5fr9q\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:45.080357 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.080333 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5fr9q\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:45.082052 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.082034 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5fr9q\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:45.087495 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.087475 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9rnv\" (UniqueName: \"kubernetes.io/projected/3a8f86d1-2385-4c9c-92e1-4561c3663f62-kube-api-access-q9rnv\") pod \"maas-api-revoke-sub-5a29efb3-5fr9q\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:45.168163 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.168090 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:45.443809 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.443785 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q"] Nov 01 22:55:45.445795 ip-10-0-128-55 kubenswrapper[6287]: W1101 22:55:45.445765 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8f86d1_2385_4c9c_92e1_4561c3663f62.slice/crio-22f5e36141f9a83775ba00bd26377a50b6ee69add31a1a0e813980c5e0bd2d32 WatchSource:0}: Error finding container 22f5e36141f9a83775ba00bd26377a50b6ee69add31a1a0e813980c5e0bd2d32: Status 404 returned error can't find the container with id 22f5e36141f9a83775ba00bd26377a50b6ee69add31a1a0e813980c5e0bd2d32 Nov 01 22:55:45.587131 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:55:45.587078 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:55:45.933212 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.933170 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" event={"ID":"3a8f86d1-2385-4c9c-92e1-4561c3663f62","Type":"ContainerStarted","Data":"cbcf22f564b0b1d022416b021ccc4226535819c5cba79987cfcf1c6e51b1ff44"} Nov 01 22:55:45.933709 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.933227 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" event={"ID":"3a8f86d1-2385-4c9c-92e1-4561c3663f62","Type":"ContainerStarted","Data":"22f5e36141f9a83775ba00bd26377a50b6ee69add31a1a0e813980c5e0bd2d32"} Nov 01 22:55:45.953433 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:45.953361 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" podStartSLOduration=1.95334435 podStartE2EDuration="1.95334435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:45.949245198 +0000 UTC m=+1958.597744436" watchObservedRunningTime="2026-11-01 22:55:45.95334435 +0000 UTC m=+1958.601843585" Nov 01 22:55:46.948609 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:46.948562 6287 generic.go:356] "Generic (PLEG): container finished" podID="3a8f86d1-2385-4c9c-92e1-4561c3663f62" containerID="cbcf22f564b0b1d022416b021ccc4226535819c5cba79987cfcf1c6e51b1ff44" exitCode=0 Nov 01 22:55:46.949287 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:46.948769 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" event={"ID":"3a8f86d1-2385-4c9c-92e1-4561c3663f62","Type":"ContainerDied","Data":"cbcf22f564b0b1d022416b021ccc4226535819c5cba79987cfcf1c6e51b1ff44"} Nov 01 22:55:48.251858 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.251594 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:55:48.347070 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.347039 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8f86d1-2385-4c9c-92e1-4561c3663f62-kube-api-access-q9rnv\" (UniqueName: \"kubernetes.io/projected/3a8f86d1-2385-4c9c-92e1-4561c3663f62-kube-api-access-q9rnv\") pod \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " Nov 01 22:55:48.347294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.347145 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-xks-ca\") pod \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " Nov 01 22:55:48.347294 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.347178 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-openshift-ca\") pod \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\" (UID: \"3a8f86d1-2385-4c9c-92e1-4561c3663f62\") " Nov 01 22:55:48.347681 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.347646 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-openshift-ca" pod "3a8f86d1-2385-4c9c-92e1-4561c3663f62" (UID: "3a8f86d1-2385-4c9c-92e1-4561c3663f62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:48.350243 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.350216 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8f86d1-2385-4c9c-92e1-4561c3663f62-kube-api-access-q9rnv" pod "3a8f86d1-2385-4c9c-92e1-4561c3663f62" (UID: "3a8f86d1-2385-4c9c-92e1-4561c3663f62"). InnerVolumeSpecName "kube-api-access-q9rnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:48.350554 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.350526 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-xks-ca" pod "3a8f86d1-2385-4c9c-92e1-4561c3663f62" (UID: "3a8f86d1-2385-4c9c-92e1-4561c3663f62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:48.448234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.448197 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9rnv\" (UniqueName: \"kubernetes.io/projected/3a8f86d1-2385-4c9c-92e1-4561c3663f62-kube-api-access-q9rnv\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:48.448234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.448224 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:48.448234 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.448234 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a8f86d1-2385-4c9c-92e1-4561c3663f62-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:48.960469 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.960438 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" event={"ID":"3a8f86d1-2385-4c9c-92e1-4561c3663f62","Type":"ContainerDied","Data":"22f5e36141f9a83775ba00bd26377a50b6ee69add31a1a0e813980c5e0bd2d32"} Nov 01 22:55:48.960646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.960483 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22f5e36141f9a83775ba00bd26377a50b6ee69add31a1a0e813980c5e0bd2d32" Nov 01 22:55:48.960646 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:55:48.960447 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q" Nov 01 22:56:15.586548 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:56:15.586509 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:56:31.974175 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:56:31.974135 6287 scope.go:122] "RemoveContainer" containerID="d7035918b6cd878d51ba5b04c8c4512c5c4cdad19edd0b1b5a3aee52657dbe68" Nov 01 22:56:31.983073 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:56:31.983050 6287 scope.go:122] "RemoveContainer" containerID="758001c1fd685931293a821e68f01f17475e37284c633b8da13f4cae6ecc1d2b" Nov 01 22:56:45.586679 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:56:45.586642 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:57:15.588183 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:57:15.588141 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:57:32.006973 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:57:32.006940 6287 scope.go:122] "RemoveContainer" containerID="f8a213231796052121ec1515e9e40e4eede7f18e13b83debb9f043fe6577bc8f" Nov 01 22:57:45.586871 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:57:45.586829 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:58:15.586523 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:58:15.586480 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:58:17.307740 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:17.307715 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 22:58:17.308388 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:17.308372 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 22:58:19.739346 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:19.739314 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 22:58:19.739990 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:19.739970 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 22:58:26.029546 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:26.029509 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s"] Nov 01 22:58:26.032786 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:26.032761 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ckv8s"] Nov 01 22:58:27.472312 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:27.472283 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ab58dc-1516-416e-b0d8-835301e9f0e2" path="/var/lib/kubelet/pods/20ab58dc-1516-416e-b0d8-835301e9f0e2/volumes" Nov 01 22:58:45.587127 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:58:45.587073 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:58:47.032367 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:47.032334 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f"] Nov 01 22:58:47.036337 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:47.036309 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-tgg4f"] Nov 01 22:58:47.469800 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:58:47.469722 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b" path="/var/lib/kubelet/pods/ac2e682c-14f8-4e86-aa6e-8b7d6b97d32b/volumes" Nov 01 22:59:05.022790 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:05.022749 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n"] Nov 01 22:59:05.025399 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:05.025376 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-fs77n"] Nov 01 22:59:05.469186 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:05.469090 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf87316-b82b-45cb-98fe-7cf529091397" path="/var/lib/kubelet/pods/7cf87316-b82b-45cb-98fe-7cf529091397/volumes" Nov 01 22:59:15.586820 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:59:15.586785 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 22:59:23.024562 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:23.024465 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7"] Nov 01 22:59:23.027237 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:23.027203 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-hf9n7"] Nov 01 22:59:23.469271 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:23.469196 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f5bc2f-df8d-4fee-8b97-33dd89874449" path="/var/lib/kubelet/pods/85f5bc2f-df8d-4fee-8b97-33dd89874449/volumes" Nov 01 22:59:32.038938 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:32.038900 6287 scope.go:122] "RemoveContainer" containerID="e4acfe786156c76ffa706858ff5e39709a25d9d4cde8362bdf1333c87a52d767" Nov 01 22:59:41.023098 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:41.023058 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b"] Nov 01 22:59:41.027161 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:41.027137 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-cbz2b"] Nov 01 22:59:41.469552 ip-10-0-128-55 kubenswrapper[6287]: I1101 22:59:41.469471 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1b74ef-2cd1-47c2-b619-c6661dbe024b" path="/var/lib/kubelet/pods/de1b74ef-2cd1-47c2-b619-c6661dbe024b/volumes" Nov 01 22:59:45.595586 ip-10-0-128-55 kubenswrapper[6287]: E1101 22:59:45.595540 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:00:00.165756 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.165717 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv"] Nov 01 23:00:00.166304 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.166100 6287 cpu_manager.go:414] "containerMap: removing container" podUID="3a8f86d1-2385-4c9c-92e1-4561c3663f62" containerName="revoke-keys" Nov 01 23:00:00.166304 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.166127 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8f86d1-2385-4c9c-92e1-4561c3663f62" containerName="revoke-keys" Nov 01 23:00:00.166304 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.166194 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8f86d1-2385-4c9c-92e1-4561c3663f62" containerName="revoke-keys" Nov 01 23:00:00.170731 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.170712 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:00.177032 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.177009 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:00:00.177155 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.177038 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:00:00.177155 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.177065 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:00:00.216164 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.216134 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv"] Nov 01 23:00:00.255019 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.254988 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bbef5-8bb1-485d-b606-db2b5d8266ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892900-hg6xv\" (UID: \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:00.255149 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.255026 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f576q\" (UniqueName: \"kubernetes.io/projected/da2bbef5-8bb1-485d-b606-db2b5d8266ce-kube-api-access-f576q\") pod \"maas-api-key-cleanup-29892900-hg6xv\" (UID: \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:00.355868 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.355835 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bbef5-8bb1-485d-b606-db2b5d8266ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892900-hg6xv\" (UID: \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:00.356024 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.355877 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f576q\" (UniqueName: \"kubernetes.io/projected/da2bbef5-8bb1-485d-b606-db2b5d8266ce-kube-api-access-f576q\") pod \"maas-api-key-cleanup-29892900-hg6xv\" (UID: \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:00.356528 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.356506 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bbef5-8bb1-485d-b606-db2b5d8266ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892900-hg6xv\" (UID: \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:00.365022 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.364999 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f576q\" (UniqueName: \"kubernetes.io/projected/da2bbef5-8bb1-485d-b606-db2b5d8266ce-kube-api-access-f576q\") pod \"maas-api-key-cleanup-29892900-hg6xv\" (UID: \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:00.484334 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.484252 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:00.756371 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:00.756339 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv"] Nov 01 23:00:00.759056 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:00:00.759028 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2bbef5_8bb1_485d_b606_db2b5d8266ce.slice/crio-c27135388ee1a36747c6003e6e77142da0d6703446e5934b9c441e2860425b91 WatchSource:0}: Error finding container c27135388ee1a36747c6003e6e77142da0d6703446e5934b9c441e2860425b91: Status 404 returned error can't find the container with id c27135388ee1a36747c6003e6e77142da0d6703446e5934b9c441e2860425b91 Nov 01 23:00:01.298108 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:01.298071 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" event={"ID":"da2bbef5-8bb1-485d-b606-db2b5d8266ce","Type":"ContainerStarted","Data":"39588aeb8dc637cbfcb9382d5e77a2af92a1c8f3af3462bde25035e161f90f9c"} Nov 01 23:00:01.298509 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:01.298143 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" event={"ID":"da2bbef5-8bb1-485d-b606-db2b5d8266ce","Type":"ContainerStarted","Data":"c27135388ee1a36747c6003e6e77142da0d6703446e5934b9c441e2860425b91"} Nov 01 23:00:01.314287 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:01.314216 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" podStartSLOduration=1.314200026 podStartE2EDuration="1.314200026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:01.314145532 +0000 UTC m=+2213.962644767" watchObservedRunningTime="2026-11-01 23:00:01.314200026 +0000 UTC m=+2213.962699263" Nov 01 23:00:02.306222 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:02.306188 6287 generic.go:356] "Generic (PLEG): container finished" podID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerID="39588aeb8dc637cbfcb9382d5e77a2af92a1c8f3af3462bde25035e161f90f9c" exitCode=60 Nov 01 23:00:02.306631 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:02.306252 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" event={"ID":"da2bbef5-8bb1-485d-b606-db2b5d8266ce","Type":"ContainerDied","Data":"39588aeb8dc637cbfcb9382d5e77a2af92a1c8f3af3462bde25035e161f90f9c"} Nov 01 23:00:02.306631 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:02.306459 6287 scope.go:122] "RemoveContainer" containerID="39588aeb8dc637cbfcb9382d5e77a2af92a1c8f3af3462bde25035e161f90f9c" Nov 01 23:00:03.313634 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:03.313603 6287 generic.go:356] "Generic (PLEG): container finished" podID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerID="5d77346ffbcd95aa12f5427851b95c20e5b734cd3a9db6bc6cc430091ac89b67" exitCode=60 Nov 01 23:00:03.314153 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:03.313688 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" event={"ID":"da2bbef5-8bb1-485d-b606-db2b5d8266ce","Type":"ContainerDied","Data":"5d77346ffbcd95aa12f5427851b95c20e5b734cd3a9db6bc6cc430091ac89b67"} Nov 01 23:00:03.314153 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:03.313747 6287 scope.go:122] "RemoveContainer" containerID="39588aeb8dc637cbfcb9382d5e77a2af92a1c8f3af3462bde25035e161f90f9c" Nov 01 23:00:03.314153 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:03.313958 6287 scope.go:122] "RemoveContainer" containerID="5d77346ffbcd95aa12f5427851b95c20e5b734cd3a9db6bc6cc430091ac89b67" Nov 01 23:00:03.314345 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:00:03.314234 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892900-hg6xv_odh-ai-gateway-infra(da2bbef5-8bb1-485d-b606-db2b5d8266ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" Nov 01 23:00:04.327970 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:04.327940 6287 scope.go:122] "RemoveContainer" containerID="5d77346ffbcd95aa12f5427851b95c20e5b734cd3a9db6bc6cc430091ac89b67" Nov 01 23:00:04.328395 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:00:04.328177 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892900-hg6xv_odh-ai-gateway-infra(da2bbef5-8bb1-485d-b606-db2b5d8266ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" Nov 01 23:00:09.029853 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:09.029814 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j"] Nov 01 23:00:09.034438 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:09.034411 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gr8j"] Nov 01 23:00:09.476253 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:09.476161 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34a7fe8-0b0a-4da6-b6a6-43b564852536" path="/var/lib/kubelet/pods/a34a7fe8-0b0a-4da6-b6a6-43b564852536/volumes" Nov 01 23:00:14.463514 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:14.463483 6287 scope.go:122] "RemoveContainer" containerID="5d77346ffbcd95aa12f5427851b95c20e5b734cd3a9db6bc6cc430091ac89b67" Nov 01 23:00:15.387049 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:15.387019 6287 generic.go:356] "Generic (PLEG): container finished" podID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerID="001eb776a45ce624163ccae0bb069bf22c49bafb600854c92acdc62aa4586002" exitCode=60 Nov 01 23:00:15.387237 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:15.387101 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" event={"ID":"da2bbef5-8bb1-485d-b606-db2b5d8266ce","Type":"ContainerDied","Data":"001eb776a45ce624163ccae0bb069bf22c49bafb600854c92acdc62aa4586002"} Nov 01 23:00:15.387237 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:15.387180 6287 scope.go:122] "RemoveContainer" containerID="5d77346ffbcd95aa12f5427851b95c20e5b734cd3a9db6bc6cc430091ac89b67" Nov 01 23:00:15.387548 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:15.387527 6287 scope.go:122] "RemoveContainer" containerID="001eb776a45ce624163ccae0bb069bf22c49bafb600854c92acdc62aa4586002" Nov 01 23:00:15.387772 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:00:15.387751 6287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892900-hg6xv_odh-ai-gateway-infra(da2bbef5-8bb1-485d-b606-db2b5d8266ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" Nov 01 23:00:15.589455 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:00:15.589418 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:00:16.415166 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:16.415130 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv"] Nov 01 23:00:16.555507 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:16.555475 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:16.703181 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:16.703071 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2bbef5-8bb1-485d-b606-db2b5d8266ce-kube-api-access-f576q\" (UniqueName: \"kubernetes.io/projected/da2bbef5-8bb1-485d-b606-db2b5d8266ce-kube-api-access-f576q\") pod \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\" (UID: \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\") " Nov 01 23:00:16.703645 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:16.703233 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da2bbef5-8bb1-485d-b606-db2b5d8266ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bbef5-8bb1-485d-b606-db2b5d8266ce-maas-api-service-ca\") pod \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\" (UID: \"da2bbef5-8bb1-485d-b606-db2b5d8266ce\") " Nov 01 23:00:16.703645 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:16.703539 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2bbef5-8bb1-485d-b606-db2b5d8266ce-maas-api-service-ca" pod "da2bbef5-8bb1-485d-b606-db2b5d8266ce" (UID: "da2bbef5-8bb1-485d-b606-db2b5d8266ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:16.705044 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:16.705021 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2bbef5-8bb1-485d-b606-db2b5d8266ce-kube-api-access-f576q" pod "da2bbef5-8bb1-485d-b606-db2b5d8266ce" (UID: "da2bbef5-8bb1-485d-b606-db2b5d8266ce"). InnerVolumeSpecName "kube-api-access-f576q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:16.804098 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:16.804069 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da2bbef5-8bb1-485d-b606-db2b5d8266ce-maas-api-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:16.804098 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:16.804092 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f576q\" (UniqueName: \"kubernetes.io/projected/da2bbef5-8bb1-485d-b606-db2b5d8266ce-kube-api-access-f576q\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:17.400931 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:17.400898 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" Nov 01 23:00:17.401107 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:17.400942 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv" event={"ID":"da2bbef5-8bb1-485d-b606-db2b5d8266ce","Type":"ContainerDied","Data":"c27135388ee1a36747c6003e6e77142da0d6703446e5934b9c441e2860425b91"} Nov 01 23:00:17.401107 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:17.401004 6287 scope.go:122] "RemoveContainer" containerID="001eb776a45ce624163ccae0bb069bf22c49bafb600854c92acdc62aa4586002" Nov 01 23:00:17.428512 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:17.428479 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv"] Nov 01 23:00:17.435355 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:17.435332 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892900-hg6xv"] Nov 01 23:00:17.468873 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:17.468843 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" path="/var/lib/kubelet/pods/da2bbef5-8bb1-485d-b606-db2b5d8266ce/volumes" Nov 01 23:00:32.061978 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:32.061943 6287 scope.go:122] "RemoveContainer" containerID="78aa489990c3b2ff5d994e5fc298027cf5e7418dcc0a62c04d1bbf2260637576" Nov 01 23:00:32.070738 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:32.070711 6287 scope.go:122] "RemoveContainer" containerID="9681cfd6ac875b41c226b9c632c6d1e05c1635eebc256d7ed33d4213e680a264" Nov 01 23:00:32.078419 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:32.078393 6287 scope.go:122] "RemoveContainer" containerID="289ef4201cce2b004b8b2c0f03b9192bf70d1522d5b0b9cc523deca1845f9861" Nov 01 23:00:38.029935 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:38.029897 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq"] Nov 01 23:00:38.032499 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:38.032475 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-p5mgq"] Nov 01 23:00:39.469248 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:39.469214 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd595b5-f707-4aad-8781-a32ab492a280" path="/var/lib/kubelet/pods/5dd595b5-f707-4aad-8781-a32ab492a280/volumes" Nov 01 23:00:45.586700 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:00:45.586659 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:00:48.022747 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:48.022665 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q"] Nov 01 23:00:48.026532 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:48.026506 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5fr9q"] Nov 01 23:00:49.468526 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:00:49.468493 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8f86d1-2385-4c9c-92e1-4561c3663f62" path="/var/lib/kubelet/pods/3a8f86d1-2385-4c9c-92e1-4561c3663f62/volumes" Nov 01 23:01:02.053856 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.053818 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x"] Nov 01 23:01:02.054285 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.054194 6287 cpu_manager.go:414] "containerMap: removing container" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerName="cleanup" Nov 01 23:01:02.054285 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.054210 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerName="cleanup" Nov 01 23:01:02.054285 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.054233 6287 cpu_manager.go:414] "containerMap: removing container" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerName="cleanup" Nov 01 23:01:02.054285 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.054238 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerName="cleanup" Nov 01 23:01:02.054285 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.054247 6287 cpu_manager.go:414] "containerMap: removing container" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerName="cleanup" Nov 01 23:01:02.054285 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.054251 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerName="cleanup" Nov 01 23:01:02.054494 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.054339 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerName="cleanup" Nov 01 23:01:02.054494 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.054352 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerName="cleanup" Nov 01 23:01:02.054494 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.054357 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2bbef5-8bb1-485d-b606-db2b5d8266ce" containerName="cleanup" Nov 01 23:01:02.057439 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.057423 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.063996 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.063953 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:01:02.065297 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.065275 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:01:02.065448 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.065410 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:01:02.065556 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.065480 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:01:02.067916 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.067892 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x"] Nov 01 23:01:02.207177 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.207146 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jln8x\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.207361 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.207199 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jln8x\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.207361 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.207220 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grtpm\" (UniqueName: \"kubernetes.io/projected/0767a8ec-0052-4f5a-8662-2253e3178346-kube-api-access-grtpm\") pod \"maas-api-revoke-sub-3d4156f1-jln8x\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.307790 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.307709 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jln8x\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.307790 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.307759 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jln8x\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.307790 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.307784 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grtpm\" (UniqueName: \"kubernetes.io/projected/0767a8ec-0052-4f5a-8662-2253e3178346-kube-api-access-grtpm\") pod \"maas-api-revoke-sub-3d4156f1-jln8x\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.308339 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.308317 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jln8x\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.310214 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.310187 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jln8x\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.316709 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.316682 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grtpm\" (UniqueName: \"kubernetes.io/projected/0767a8ec-0052-4f5a-8662-2253e3178346-kube-api-access-grtpm\") pod \"maas-api-revoke-sub-3d4156f1-jln8x\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.371866 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.371827 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:02.643481 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:02.643449 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x"] Nov 01 23:01:02.646359 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:01:02.646323 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0767a8ec_0052_4f5a_8662_2253e3178346.slice/crio-c3383ba628dfd2d4342cb349655a0f83df4241f29f3ce1d9af87ecac8cdc7902 WatchSource:0}: Error finding container c3383ba628dfd2d4342cb349655a0f83df4241f29f3ce1d9af87ecac8cdc7902: Status 404 returned error can't find the container with id c3383ba628dfd2d4342cb349655a0f83df4241f29f3ce1d9af87ecac8cdc7902 Nov 01 23:01:03.642125 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:03.642077 6287 generic.go:356] "Generic (PLEG): container finished" podID="0767a8ec-0052-4f5a-8662-2253e3178346" containerID="5e56a75affd3283e0447c3a2d18ea74d6bd6799aa46060b68281dd76cd4ac53b" exitCode=0 Nov 01 23:01:03.642528 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:03.642162 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" event={"ID":"0767a8ec-0052-4f5a-8662-2253e3178346","Type":"ContainerDied","Data":"5e56a75affd3283e0447c3a2d18ea74d6bd6799aa46060b68281dd76cd4ac53b"} Nov 01 23:01:03.642528 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:03.642217 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" event={"ID":"0767a8ec-0052-4f5a-8662-2253e3178346","Type":"ContainerStarted","Data":"c3383ba628dfd2d4342cb349655a0f83df4241f29f3ce1d9af87ecac8cdc7902"} Nov 01 23:01:04.780072 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:04.780048 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:04.932721 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:04.932638 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-openshift-ca\") pod \"0767a8ec-0052-4f5a-8662-2253e3178346\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " Nov 01 23:01:04.932880 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:04.932830 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0767a8ec-0052-4f5a-8662-2253e3178346-kube-api-access-grtpm\" (UniqueName: \"kubernetes.io/projected/0767a8ec-0052-4f5a-8662-2253e3178346-kube-api-access-grtpm\") pod \"0767a8ec-0052-4f5a-8662-2253e3178346\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " Nov 01 23:01:04.932945 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:04.932886 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-xks-ca\") pod \"0767a8ec-0052-4f5a-8662-2253e3178346\" (UID: \"0767a8ec-0052-4f5a-8662-2253e3178346\") " Nov 01 23:01:04.932999 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:04.932973 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-openshift-ca" pod "0767a8ec-0052-4f5a-8662-2253e3178346" (UID: "0767a8ec-0052-4f5a-8662-2253e3178346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:04.933136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:04.933099 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:04.934825 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:04.934798 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0767a8ec-0052-4f5a-8662-2253e3178346-kube-api-access-grtpm" pod "0767a8ec-0052-4f5a-8662-2253e3178346" (UID: "0767a8ec-0052-4f5a-8662-2253e3178346"). InnerVolumeSpecName "kube-api-access-grtpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:04.934927 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:04.934844 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-xks-ca" pod "0767a8ec-0052-4f5a-8662-2253e3178346" (UID: "0767a8ec-0052-4f5a-8662-2253e3178346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:01:05.033818 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:05.033787 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grtpm\" (UniqueName: \"kubernetes.io/projected/0767a8ec-0052-4f5a-8662-2253e3178346-kube-api-access-grtpm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:05.033818 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:05.033815 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0767a8ec-0052-4f5a-8662-2253e3178346-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:05.653788 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:05.653757 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" Nov 01 23:01:05.653788 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:05.653780 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x" event={"ID":"0767a8ec-0052-4f5a-8662-2253e3178346","Type":"ContainerDied","Data":"c3383ba628dfd2d4342cb349655a0f83df4241f29f3ce1d9af87ecac8cdc7902"} Nov 01 23:01:05.654005 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:05.653824 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3383ba628dfd2d4342cb349655a0f83df4241f29f3ce1d9af87ecac8cdc7902" Nov 01 23:01:10.379678 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:10.379644 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds"] Nov 01 23:01:10.380218 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:10.379865 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" containerID="cri-o://bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5" gracePeriod=30 Nov 01 23:01:15.603326 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:01:15.603279 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:01:15.646384 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.646363 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 23:01:15.655636 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655617 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-istio-token\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-istio-token\") pod \"71935876-95e7-4691-bedf-e040f4cfec12\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " Nov 01 23:01:15.655713 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655649 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71935876-95e7-4691-bedf-e040f4cfec12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71935876-95e7-4691-bedf-e040f4cfec12-istio-podinfo\") pod \"71935876-95e7-4691-bedf-e040f4cfec12\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " Nov 01 23:01:15.655713 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655686 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-socket\") pod \"71935876-95e7-4691-bedf-e040f4cfec12\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " Nov 01 23:01:15.655713 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655702 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-kube-api-access-kc7pb\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-kube-api-access-kc7pb\") pod \"71935876-95e7-4691-bedf-e040f4cfec12\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " Nov 01 23:01:15.655838 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655744 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-certs\") pod \"71935876-95e7-4691-bedf-e040f4cfec12\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " Nov 01 23:01:15.655838 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655762 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-envoy\") pod \"71935876-95e7-4691-bedf-e040f4cfec12\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " Nov 01 23:01:15.655838 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655793 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71935876-95e7-4691-bedf-e040f4cfec12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71935876-95e7-4691-bedf-e040f4cfec12-istiod-ca-cert\") pod \"71935876-95e7-4691-bedf-e040f4cfec12\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " Nov 01 23:01:15.655838 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655822 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-credential-socket\") pod \"71935876-95e7-4691-bedf-e040f4cfec12\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " Nov 01 23:01:15.655998 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655868 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-data\") pod \"71935876-95e7-4691-bedf-e040f4cfec12\" (UID: \"71935876-95e7-4691-bedf-e040f4cfec12\") " Nov 01 23:01:15.655998 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.655966 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-socket" pod "71935876-95e7-4691-bedf-e040f4cfec12" (UID: "71935876-95e7-4691-bedf-e040f4cfec12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.656194 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.656179 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.656568 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.656529 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-certs" pod "71935876-95e7-4691-bedf-e040f4cfec12" (UID: "71935876-95e7-4691-bedf-e040f4cfec12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.656748 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.656731 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-data" pod "71935876-95e7-4691-bedf-e040f4cfec12" (UID: "71935876-95e7-4691-bedf-e040f4cfec12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.656834 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.656728 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-credential-socket" pod "71935876-95e7-4691-bedf-e040f4cfec12" (UID: "71935876-95e7-4691-bedf-e040f4cfec12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.657216 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.657192 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71935876-95e7-4691-bedf-e040f4cfec12-istiod-ca-cert" pod "71935876-95e7-4691-bedf-e040f4cfec12" (UID: "71935876-95e7-4691-bedf-e040f4cfec12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:15.657945 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.657920 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71935876-95e7-4691-bedf-e040f4cfec12-istio-podinfo" pod "71935876-95e7-4691-bedf-e040f4cfec12" (UID: "71935876-95e7-4691-bedf-e040f4cfec12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:15.658571 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.658549 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-istio-token" pod "71935876-95e7-4691-bedf-e040f4cfec12" (UID: "71935876-95e7-4691-bedf-e040f4cfec12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.658828 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.658810 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-envoy" pod "71935876-95e7-4691-bedf-e040f4cfec12" (UID: "71935876-95e7-4691-bedf-e040f4cfec12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:15.658895 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.658866 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-kube-api-access-kc7pb" pod "71935876-95e7-4691-bedf-e040f4cfec12" (UID: "71935876-95e7-4691-bedf-e040f4cfec12"). InnerVolumeSpecName "kube-api-access-kc7pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:15.749193 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.749158 6287 generic.go:356] "Generic (PLEG): container finished" podID="71935876-95e7-4691-bedf-e040f4cfec12" containerID="bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5" exitCode=0 Nov 01 23:01:15.749383 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.749248 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" event={"ID":"71935876-95e7-4691-bedf-e040f4cfec12","Type":"ContainerDied","Data":"bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5"} Nov 01 23:01:15.749383 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.749318 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" event={"ID":"71935876-95e7-4691-bedf-e040f4cfec12","Type":"ContainerDied","Data":"e25e0fc34a38e8a8d7546d4a4b592f75731291638e10dea5dd255f437d599434"} Nov 01 23:01:15.749383 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.749346 6287 scope.go:122] "RemoveContainer" containerID="bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5" Nov 01 23:01:15.749536 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.749236 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds" Nov 01 23:01:15.756618 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.756580 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc7pb\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-kube-api-access-kc7pb\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.756618 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.756604 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.756618 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.756615 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.756898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.756625 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71935876-95e7-4691-bedf-e040f4cfec12-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.756898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.756633 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.756898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.756643 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71935876-95e7-4691-bedf-e040f4cfec12-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.756898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.756651 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71935876-95e7-4691-bedf-e040f4cfec12-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.756898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.756659 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71935876-95e7-4691-bedf-e040f4cfec12-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:15.763605 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.763587 6287 scope.go:122] "RemoveContainer" containerID="2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c" Nov 01 23:01:15.777872 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.777850 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds"] Nov 01 23:01:15.780625 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.780602 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-87hds"] Nov 01 23:01:15.812952 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.812925 6287 scope.go:122] "RemoveContainer" containerID="bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5" Nov 01 23:01:15.813250 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:01:15.813229 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5\": container with ID starting with bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5 not found: ID does not exist" containerID="bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5" Nov 01 23:01:15.813349 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.813260 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5"} err="failed to get container status \"bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5\": rpc error: code = NotFound desc = could not find container \"bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5\": container with ID starting with bed14f477b8eba32d6134c4f4fa1047795ca95df4a3f92af5655ff40c9dfdbd5 not found: ID does not exist" Nov 01 23:01:15.813349 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.813281 6287 scope.go:122] "RemoveContainer" containerID="2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c" Nov 01 23:01:15.813597 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:01:15.813577 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c\": container with ID starting with 2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c not found: ID does not exist" containerID="2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c" Nov 01 23:01:15.813649 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:15.813603 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c"} err="failed to get container status \"2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c\": rpc error: code = NotFound desc = could not find container \"2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c\": container with ID starting with 2adfb92d8428c9a98c24d37a1a6381eca6f10d9360c86a9f2bd22d574445464c not found: ID does not exist" Nov 01 23:01:17.469838 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:17.469801 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71935876-95e7-4691-bedf-e040f4cfec12" path="/var/lib/kubelet/pods/71935876-95e7-4691-bedf-e040f4cfec12/volumes" Nov 01 23:01:25.417294 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.417250 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c"] Nov 01 23:01:25.417861 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.417839 6287 cpu_manager.go:414] "containerMap: removing container" podUID="0767a8ec-0052-4f5a-8662-2253e3178346" containerName="revoke-keys" Nov 01 23:01:25.417952 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.417863 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0767a8ec-0052-4f5a-8662-2253e3178346" containerName="revoke-keys" Nov 01 23:01:25.417952 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.417881 6287 cpu_manager.go:414] "containerMap: removing container" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" Nov 01 23:01:25.417952 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.417889 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" Nov 01 23:01:25.417952 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.417914 6287 cpu_manager.go:414] "containerMap: removing container" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" Nov 01 23:01:25.417952 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.417922 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" Nov 01 23:01:25.418258 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.418008 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0767a8ec-0052-4f5a-8662-2253e3178346" containerName="revoke-keys" Nov 01 23:01:25.418258 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.418022 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" Nov 01 23:01:25.418258 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.418034 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="71935876-95e7-4691-bedf-e040f4cfec12" containerName="istio-proxy" Nov 01 23:01:25.422528 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.422505 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:25.433915 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.433889 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c"] Nov 01 23:01:25.439167 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.439141 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhm4s\" (UniqueName: \"kubernetes.io/projected/8e8604a4-2f6e-4212-84b0-6320b25ebca7-kube-api-access-jhm4s\") pod \"payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c\" (UID: \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:25.439286 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.439217 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8604a4-2f6e-4212-84b0-6320b25ebca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c\" (UID: \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:25.458461 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.458427 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7ae63f\"" Nov 01 23:01:25.470889 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.470863 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m"] Nov 01 23:01:25.474674 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.474652 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:25.491399 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.491374 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m"] Nov 01 23:01:25.540609 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.540574 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhm4s\" (UniqueName: \"kubernetes.io/projected/8e8604a4-2f6e-4212-84b0-6320b25ebca7-kube-api-access-jhm4s\") pod \"payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c\" (UID: \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:25.540771 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.540621 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b23808-f190-4ecd-bfca-c0e1fba243b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m\" (UID: \"40b23808-f190-4ecd-bfca-c0e1fba243b3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:25.540771 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.540655 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8604a4-2f6e-4212-84b0-6320b25ebca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c\" (UID: \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:25.540771 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.540673 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2vpw\" (UniqueName: \"kubernetes.io/projected/40b23808-f190-4ecd-bfca-c0e1fba243b3-kube-api-access-z2vpw\") pod \"payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m\" (UID: \"40b23808-f190-4ecd-bfca-c0e1fba243b3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:25.541410 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.541392 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8604a4-2f6e-4212-84b0-6320b25ebca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c\" (UID: \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:25.550089 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.550065 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhm4s\" (UniqueName: \"kubernetes.io/projected/8e8604a4-2f6e-4212-84b0-6320b25ebca7-kube-api-access-jhm4s\") pod \"payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c\" (UID: \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:25.557508 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.557487 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-40b668af-dockercfg-d49fz\"" Nov 01 23:01:25.642081 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.642042 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b23808-f190-4ecd-bfca-c0e1fba243b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m\" (UID: \"40b23808-f190-4ecd-bfca-c0e1fba243b3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:25.642286 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.642126 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vpw\" (UniqueName: \"kubernetes.io/projected/40b23808-f190-4ecd-bfca-c0e1fba243b3-kube-api-access-z2vpw\") pod \"payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m\" (UID: \"40b23808-f190-4ecd-bfca-c0e1fba243b3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:25.642788 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.642766 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b23808-f190-4ecd-bfca-c0e1fba243b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m\" (UID: \"40b23808-f190-4ecd-bfca-c0e1fba243b3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:25.651227 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.651202 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vpw\" (UniqueName: \"kubernetes.io/projected/40b23808-f190-4ecd-bfca-c0e1fba243b3-kube-api-access-z2vpw\") pod \"payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m\" (UID: \"40b23808-f190-4ecd-bfca-c0e1fba243b3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:25.737982 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.737953 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:25.787808 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:25.787668 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:26.015384 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.015356 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c"] Nov 01 23:01:26.016629 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:01:26.016604 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8604a4_2f6e_4212_84b0_6320b25ebca7.slice/crio-f352d9d067cfe7bb7fce1bcfae50d65af50b4bc655725afc9b3b84f26f8229d1 WatchSource:0}: Error finding container f352d9d067cfe7bb7fce1bcfae50d65af50b4bc655725afc9b3b84f26f8229d1: Status 404 returned error can't find the container with id f352d9d067cfe7bb7fce1bcfae50d65af50b4bc655725afc9b3b84f26f8229d1 Nov 01 23:01:26.091160 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.091102 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m"] Nov 01 23:01:26.093664 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:01:26.093638 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b23808_f190_4ecd_bfca_c0e1fba243b3.slice/crio-fcf0070694438b2d7b7d5dfec291d4100ea0b7a3b2065476c09387635608c8a9 WatchSource:0}: Error finding container fcf0070694438b2d7b7d5dfec291d4100ea0b7a3b2065476c09387635608c8a9: Status 404 returned error can't find the container with id fcf0070694438b2d7b7d5dfec291d4100ea0b7a3b2065476c09387635608c8a9 Nov 01 23:01:26.815142 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.815087 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" event={"ID":"8e8604a4-2f6e-4212-84b0-6320b25ebca7","Type":"ContainerStarted","Data":"4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb"} Nov 01 23:01:26.815667 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.815172 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:26.815667 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.815196 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" event={"ID":"8e8604a4-2f6e-4212-84b0-6320b25ebca7","Type":"ContainerStarted","Data":"f352d9d067cfe7bb7fce1bcfae50d65af50b4bc655725afc9b3b84f26f8229d1"} Nov 01 23:01:26.816777 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.816747 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" event={"ID":"40b23808-f190-4ecd-bfca-c0e1fba243b3","Type":"ContainerStarted","Data":"373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa"} Nov 01 23:01:26.816871 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.816802 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" event={"ID":"40b23808-f190-4ecd-bfca-c0e1fba243b3","Type":"ContainerStarted","Data":"fcf0070694438b2d7b7d5dfec291d4100ea0b7a3b2065476c09387635608c8a9"} Nov 01 23:01:26.816871 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.816858 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:26.835741 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.835694 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" podStartSLOduration=1.835677617 podStartE2EDuration="1.835677617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:26.834810468 +0000 UTC m=+2299.483309705" watchObservedRunningTime="2026-11-01 23:01:26.835677617 +0000 UTC m=+2299.484176901" Nov 01 23:01:26.856735 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:26.856682 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" podStartSLOduration=1.8566688839999999 podStartE2EDuration="1.856668884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:26.855767114 +0000 UTC m=+2299.504266352" watchObservedRunningTime="2026-11-01 23:01:26.856668884 +0000 UTC m=+2299.505168112" Nov 01 23:01:32.109550 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:32.109516 6287 scope.go:122] "RemoveContainer" containerID="cbd7ce6df76657fa723bac14890eae7a9cff8e21ab11d12a9a61ffb713767a1e" Nov 01 23:01:32.118436 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:32.118413 6287 scope.go:122] "RemoveContainer" containerID="1317388fe5d61480a0e0585eca4e3e90894880dbc12ef43cdcab220ad0e4d278" Nov 01 23:01:32.126708 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:32.126685 6287 scope.go:122] "RemoveContainer" containerID="17a73c8053f3410f5fba4872ff45d7fc0cd531b97eeca4964705a2ba93234387" Nov 01 23:01:37.825712 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:37.825682 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:01:37.826341 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:37.825843 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:01:41.831257 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.831220 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v"] Nov 01 23:01:41.836639 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.836614 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:41.841609 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.841582 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7ae63f\"/\"kube-root-ca.crt\"" Nov 01 23:01:41.844944 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.844922 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7ae63f\"/\"openshift-service-ca.crt\"" Nov 01 23:01:41.846869 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.846843 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7ae63f\"/\"simulator-single-main-7ae63f-kserve-self-signed-certs\"" Nov 01 23:01:41.847036 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.847001 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7ae63f\"/\"default-dockercfg-j626h\"" Nov 01 23:01:41.848648 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.848625 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v"] Nov 01 23:01:41.987856 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.987818 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-home\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:41.988038 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.987872 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tls-certs\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:41.988038 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.987998 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tmp-dir\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:41.988175 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.988136 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-model-cache\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:41.988230 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.988199 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmn9l\" (UniqueName: \"kubernetes.io/projected/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-kube-api-access-jmn9l\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:41.988301 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:41.988275 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-dshm\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.089500 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.089416 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-dshm\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.089500 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.089466 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-home\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.089500 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.089486 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tls-certs\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.089801 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.089507 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tmp-dir\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.089801 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.089544 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-model-cache\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.089801 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.089575 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmn9l\" (UniqueName: \"kubernetes.io/projected/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-kube-api-access-jmn9l\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.090002 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.089971 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-home\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.090145 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.090014 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tmp-dir\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.090145 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.090055 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-model-cache\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.091870 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.091842 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-dshm\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.092159 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.092136 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tls-certs\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.097072 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.097051 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmn9l\" (UniqueName: \"kubernetes.io/projected/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-kube-api-access-jmn9l\") pod \"simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.150725 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.150700 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:42.424568 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.424535 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v"] Nov 01 23:01:42.427595 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:01:42.427554 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb310e4f_c5e9_41d9_abee_b3e2ececb2fe.slice/crio-6445f6c7bf0914c39d87a977963b19c7dde3954562f192ef10d23a887143781c WatchSource:0}: Error finding container 6445f6c7bf0914c39d87a977963b19c7dde3954562f192ef10d23a887143781c: Status 404 returned error can't find the container with id 6445f6c7bf0914c39d87a977963b19c7dde3954562f192ef10d23a887143781c Nov 01 23:01:42.903956 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.903922 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" event={"ID":"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe","Type":"ContainerStarted","Data":"d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56"} Nov 01 23:01:42.904386 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.903966 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" event={"ID":"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe","Type":"ContainerStarted","Data":"6445f6c7bf0914c39d87a977963b19c7dde3954562f192ef10d23a887143781c"} Nov 01 23:01:42.921049 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:42.921000 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" podStartSLOduration=1.9209877180000001 podStartE2EDuration="1.920987718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:42.920706322 +0000 UTC m=+2315.569205557" watchObservedRunningTime="2026-11-01 23:01:42.920987718 +0000 UTC m=+2315.569486953" Nov 01 23:01:43.151809 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:43.151768 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:45.587027 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:01:45.586989 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:01:52.151032 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:52.150992 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:52.163872 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:52.163847 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:01:52.964644 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:01:52.964616 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:02:10.775860 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.775823 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv"] Nov 01 23:02:10.779299 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.779282 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:10.788805 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.788771 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:02:10.788972 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.788952 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:02:10.789079 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.789055 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:02:10.789267 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.789137 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:02:10.791731 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.791707 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv"] Nov 01 23:02:10.852956 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.852922 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18448dea-q9jjv\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:10.853094 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.853027 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2c69\" (UniqueName: \"kubernetes.io/projected/455b0b80-8bf6-4709-8276-d58ec2fc26db-kube-api-access-x2c69\") pod \"maas-api-revoke-sub-18448dea-q9jjv\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:10.853094 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.853070 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18448dea-q9jjv\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:10.954315 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.954282 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18448dea-q9jjv\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:10.954507 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.954350 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2c69\" (UniqueName: \"kubernetes.io/projected/455b0b80-8bf6-4709-8276-d58ec2fc26db-kube-api-access-x2c69\") pod \"maas-api-revoke-sub-18448dea-q9jjv\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:10.954507 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.954387 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18448dea-q9jjv\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:10.955076 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.955049 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18448dea-q9jjv\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:10.956768 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.956747 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18448dea-q9jjv\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:10.962708 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:10.962688 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2c69\" (UniqueName: \"kubernetes.io/projected/455b0b80-8bf6-4709-8276-d58ec2fc26db-kube-api-access-x2c69\") pod \"maas-api-revoke-sub-18448dea-q9jjv\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:11.093761 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:11.093674 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:11.364781 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:11.364751 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv"] Nov 01 23:02:11.367413 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:02:11.367378 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455b0b80_8bf6_4709_8276_d58ec2fc26db.slice/crio-fe5827f4e2a16e63355b6587b7df354b9c2ffea945ef2a6d92ec7272c495494d WatchSource:0}: Error finding container fe5827f4e2a16e63355b6587b7df354b9c2ffea945ef2a6d92ec7272c495494d: Status 404 returned error can't find the container with id fe5827f4e2a16e63355b6587b7df354b9c2ffea945ef2a6d92ec7272c495494d Nov 01 23:02:12.065913 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:12.065874 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" event={"ID":"455b0b80-8bf6-4709-8276-d58ec2fc26db","Type":"ContainerStarted","Data":"c026d32ff59a4a0ced0b61177fbc94444fa370aa6daea5ddc31cda3333729115"} Nov 01 23:02:12.066430 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:12.065927 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" event={"ID":"455b0b80-8bf6-4709-8276-d58ec2fc26db","Type":"ContainerStarted","Data":"fe5827f4e2a16e63355b6587b7df354b9c2ffea945ef2a6d92ec7272c495494d"} Nov 01 23:02:12.084267 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:12.084220 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" podStartSLOduration=2.084206734 podStartE2EDuration="2.084206734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:12.082056212 +0000 UTC m=+2344.730555448" watchObservedRunningTime="2026-11-01 23:02:12.084206734 +0000 UTC m=+2344.732705968" Nov 01 23:02:13.072830 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:13.072793 6287 generic.go:356] "Generic (PLEG): container finished" podID="455b0b80-8bf6-4709-8276-d58ec2fc26db" containerID="c026d32ff59a4a0ced0b61177fbc94444fa370aa6daea5ddc31cda3333729115" exitCode=0 Nov 01 23:02:13.073307 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:13.072840 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" event={"ID":"455b0b80-8bf6-4709-8276-d58ec2fc26db","Type":"ContainerDied","Data":"c026d32ff59a4a0ced0b61177fbc94444fa370aa6daea5ddc31cda3333729115"} Nov 01 23:02:14.210676 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.210650 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:14.282784 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.282748 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455b0b80-8bf6-4709-8276-d58ec2fc26db-kube-api-access-x2c69\" (UniqueName: \"kubernetes.io/projected/455b0b80-8bf6-4709-8276-d58ec2fc26db-kube-api-access-x2c69\") pod \"455b0b80-8bf6-4709-8276-d58ec2fc26db\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " Nov 01 23:02:14.282984 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.282813 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-xks-ca\") pod \"455b0b80-8bf6-4709-8276-d58ec2fc26db\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " Nov 01 23:02:14.282984 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.282854 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-openshift-ca\") pod \"455b0b80-8bf6-4709-8276-d58ec2fc26db\" (UID: \"455b0b80-8bf6-4709-8276-d58ec2fc26db\") " Nov 01 23:02:14.283316 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.283284 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-openshift-ca" pod "455b0b80-8bf6-4709-8276-d58ec2fc26db" (UID: "455b0b80-8bf6-4709-8276-d58ec2fc26db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:14.284960 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.284938 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455b0b80-8bf6-4709-8276-d58ec2fc26db-kube-api-access-x2c69" pod "455b0b80-8bf6-4709-8276-d58ec2fc26db" (UID: "455b0b80-8bf6-4709-8276-d58ec2fc26db"). InnerVolumeSpecName "kube-api-access-x2c69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:14.285065 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.285030 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-xks-ca" pod "455b0b80-8bf6-4709-8276-d58ec2fc26db" (UID: "455b0b80-8bf6-4709-8276-d58ec2fc26db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:14.384415 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.384327 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2c69\" (UniqueName: \"kubernetes.io/projected/455b0b80-8bf6-4709-8276-d58ec2fc26db-kube-api-access-x2c69\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:14.384415 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.384361 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:14.384415 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:14.384372 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/455b0b80-8bf6-4709-8276-d58ec2fc26db-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:15.083937 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:15.083911 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" Nov 01 23:02:15.084130 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:15.083900 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv" event={"ID":"455b0b80-8bf6-4709-8276-d58ec2fc26db","Type":"ContainerDied","Data":"fe5827f4e2a16e63355b6587b7df354b9c2ffea945ef2a6d92ec7272c495494d"} Nov 01 23:02:15.084130 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:15.084050 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe5827f4e2a16e63355b6587b7df354b9c2ffea945ef2a6d92ec7272c495494d" Nov 01 23:02:15.587272 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:02:15.587227 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:02:21.872165 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.872075 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s"] Nov 01 23:02:21.872548 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.872468 6287 cpu_manager.go:414] "containerMap: removing container" podUID="455b0b80-8bf6-4709-8276-d58ec2fc26db" containerName="revoke-keys" Nov 01 23:02:21.872548 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.872481 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455b0b80-8bf6-4709-8276-d58ec2fc26db" containerName="revoke-keys" Nov 01 23:02:21.872627 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.872553 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="455b0b80-8bf6-4709-8276-d58ec2fc26db" containerName="revoke-keys" Nov 01 23:02:21.876121 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.876088 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:21.884485 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.884260 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s"] Nov 01 23:02:21.884585 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.884494 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:02:21.884585 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.884509 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:02:21.885526 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.885504 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:02:21.885642 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.885547 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:02:21.953329 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.953287 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pcqj\" (UniqueName: \"kubernetes.io/projected/da63655d-f967-49fe-8ecb-70090a538184-kube-api-access-8pcqj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:21.953519 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.953353 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da63655d-f967-49fe-8ecb-70090a538184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:21.953519 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:21.953492 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da63655d-f967-49fe-8ecb-70090a538184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:22.054579 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:22.054545 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pcqj\" (UniqueName: \"kubernetes.io/projected/da63655d-f967-49fe-8ecb-70090a538184-kube-api-access-8pcqj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:22.054737 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:22.054587 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da63655d-f967-49fe-8ecb-70090a538184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:22.054737 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:22.054608 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da63655d-f967-49fe-8ecb-70090a538184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:22.055239 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:22.055221 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da63655d-f967-49fe-8ecb-70090a538184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:22.056964 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:22.056935 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da63655d-f967-49fe-8ecb-70090a538184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:22.062184 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:22.062160 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pcqj\" (UniqueName: \"kubernetes.io/projected/da63655d-f967-49fe-8ecb-70090a538184-kube-api-access-8pcqj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:22.191508 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:22.191439 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:22.466263 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:22.466236 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s"] Nov 01 23:02:22.468728 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:02:22.468701 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda63655d_f967_49fe_8ecb_70090a538184.slice/crio-62be7acec84bef38ae289d779a58a16275491123e35086f6ea003edd810fad2e WatchSource:0}: Error finding container 62be7acec84bef38ae289d779a58a16275491123e35086f6ea003edd810fad2e: Status 404 returned error can't find the container with id 62be7acec84bef38ae289d779a58a16275491123e35086f6ea003edd810fad2e Nov 01 23:02:23.140831 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:23.140743 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" event={"ID":"da63655d-f967-49fe-8ecb-70090a538184","Type":"ContainerStarted","Data":"3103462df0acf863ab3629383d0b2052b3d22f4641611d3502f2b34ce2cf4723"} Nov 01 23:02:23.140831 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:23.140816 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" event={"ID":"da63655d-f967-49fe-8ecb-70090a538184","Type":"ContainerStarted","Data":"62be7acec84bef38ae289d779a58a16275491123e35086f6ea003edd810fad2e"} Nov 01 23:02:23.157245 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:23.157186 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" podStartSLOduration=2.1571693769999998 podStartE2EDuration="2.157169377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:23.156629693 +0000 UTC m=+2355.805128930" watchObservedRunningTime="2026-11-01 23:02:23.157169377 +0000 UTC m=+2355.805668612" Nov 01 23:02:24.148174 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:24.148136 6287 generic.go:356] "Generic (PLEG): container finished" podID="da63655d-f967-49fe-8ecb-70090a538184" containerID="3103462df0acf863ab3629383d0b2052b3d22f4641611d3502f2b34ce2cf4723" exitCode=0 Nov 01 23:02:24.148595 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:24.148211 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" event={"ID":"da63655d-f967-49fe-8ecb-70090a538184","Type":"ContainerDied","Data":"3103462df0acf863ab3629383d0b2052b3d22f4641611d3502f2b34ce2cf4723"} Nov 01 23:02:25.292335 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.292310 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:25.385989 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.385960 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da63655d-f967-49fe-8ecb-70090a538184-kube-api-access-8pcqj\" (UniqueName: \"kubernetes.io/projected/da63655d-f967-49fe-8ecb-70090a538184-kube-api-access-8pcqj\") pod \"da63655d-f967-49fe-8ecb-70090a538184\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " Nov 01 23:02:25.386194 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.385999 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da63655d-f967-49fe-8ecb-70090a538184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da63655d-f967-49fe-8ecb-70090a538184-maas-api-openshift-ca\") pod \"da63655d-f967-49fe-8ecb-70090a538184\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " Nov 01 23:02:25.386194 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.386137 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da63655d-f967-49fe-8ecb-70090a538184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da63655d-f967-49fe-8ecb-70090a538184-maas-api-xks-ca\") pod \"da63655d-f967-49fe-8ecb-70090a538184\" (UID: \"da63655d-f967-49fe-8ecb-70090a538184\") " Nov 01 23:02:25.386471 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.386447 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da63655d-f967-49fe-8ecb-70090a538184-maas-api-openshift-ca" pod "da63655d-f967-49fe-8ecb-70090a538184" (UID: "da63655d-f967-49fe-8ecb-70090a538184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:25.388125 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.388082 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da63655d-f967-49fe-8ecb-70090a538184-maas-api-xks-ca" pod "da63655d-f967-49fe-8ecb-70090a538184" (UID: "da63655d-f967-49fe-8ecb-70090a538184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:25.388243 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.388151 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da63655d-f967-49fe-8ecb-70090a538184-kube-api-access-8pcqj" pod "da63655d-f967-49fe-8ecb-70090a538184" (UID: "da63655d-f967-49fe-8ecb-70090a538184"). InnerVolumeSpecName "kube-api-access-8pcqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:25.487091 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.487066 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da63655d-f967-49fe-8ecb-70090a538184-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.487221 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.487091 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pcqj\" (UniqueName: \"kubernetes.io/projected/da63655d-f967-49fe-8ecb-70090a538184-kube-api-access-8pcqj\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.487221 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:25.487108 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da63655d-f967-49fe-8ecb-70090a538184-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:26.160650 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:26.160620 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" Nov 01 23:02:26.160912 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:26.160619 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s" event={"ID":"da63655d-f967-49fe-8ecb-70090a538184","Type":"ContainerDied","Data":"62be7acec84bef38ae289d779a58a16275491123e35086f6ea003edd810fad2e"} Nov 01 23:02:26.160912 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:26.160752 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62be7acec84bef38ae289d779a58a16275491123e35086f6ea003edd810fad2e" Nov 01 23:02:26.689296 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:26.689248 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v"] Nov 01 23:02:32.012530 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:32.012491 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m"] Nov 01 23:02:32.034188 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:32.034149 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c"] Nov 01 23:02:32.163511 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:32.163479 6287 scope.go:122] "RemoveContainer" containerID="cbcf22f564b0b1d022416b021ccc4226535819c5cba79987cfcf1c6e51b1ff44" Nov 01 23:02:32.172485 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:32.172465 6287 scope.go:122] "RemoveContainer" containerID="5d1bb4a7dc95fd67bd13c564cb65db1fe431c7538f2ee632a94638cab2333670" Nov 01 23:02:33.971732 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:33.971696 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s"] Nov 01 23:02:33.974789 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:33.974759 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ae63f-jx62s"] Nov 01 23:02:35.470284 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:35.470247 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da63655d-f967-49fe-8ecb-70090a538184" path="/var/lib/kubelet/pods/da63655d-f967-49fe-8ecb-70090a538184/volumes" Nov 01 23:02:37.013191 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:37.013152 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" podUID="40b23808-f190-4ecd-bfca-c0e1fba243b3" containerName="payload-processing" containerID="cri-o://373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa" gracePeriod=25 Nov 01 23:02:37.035522 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:37.035486 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" podUID="8e8604a4-2f6e-4212-84b0-6320b25ebca7" containerName="payload-pre-processing" containerID="cri-o://4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb" gracePeriod=25 Nov 01 23:02:37.823821 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:37.823780 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" podUID="40b23808-f190-4ecd-bfca-c0e1fba243b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:02:37.824021 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:37.823787 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" podUID="8e8604a4-2f6e-4212-84b0-6320b25ebca7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:02:41.710606 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:41.710536 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" podUID="bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" containerName="main" containerID="cri-o://d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56" gracePeriod=45 Nov 01 23:02:41.973953 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:41.973889 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:02:42.038083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.038045 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x"] Nov 01 23:02:42.044814 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.044784 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jln8x"] Nov 01 23:02:42.068086 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.068052 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6"] Nov 01 23:02:42.068809 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.068782 6287 cpu_manager.go:414] "containerMap: removing container" podUID="bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" containerName="main" Nov 01 23:02:42.068869 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.068815 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" containerName="main" Nov 01 23:02:42.068869 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.068858 6287 cpu_manager.go:414] "containerMap: removing container" podUID="da63655d-f967-49fe-8ecb-70090a538184" containerName="revoke-keys" Nov 01 23:02:42.069000 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.068866 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da63655d-f967-49fe-8ecb-70090a538184" containerName="revoke-keys" Nov 01 23:02:42.069078 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.069015 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="da63655d-f967-49fe-8ecb-70090a538184" containerName="revoke-keys" Nov 01 23:02:42.069078 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.069039 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" containerName="main" Nov 01 23:02:42.072669 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.072652 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.078381 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.078355 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6"] Nov 01 23:02:42.078655 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.078634 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:02:42.078885 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.078866 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:02:42.079047 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.079027 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:02:42.079264 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.079247 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:02:42.138834 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.138791 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-kube-api-access-jmn9l\" (UniqueName: \"kubernetes.io/projected/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-kube-api-access-jmn9l\") pod \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " Nov 01 23:02:42.139008 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.138881 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tmp-dir\") pod \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " Nov 01 23:02:42.139008 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.138937 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-dshm\") pod \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " Nov 01 23:02:42.139008 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.138996 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-home\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-home\") pod \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " Nov 01 23:02:42.139213 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139022 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tls-certs\") pod \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " Nov 01 23:02:42.139213 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139051 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-model-cache\") pod \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\" (UID: \"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe\") " Nov 01 23:02:42.139318 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139195 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tmp-dir" pod "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" (UID: "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:42.139318 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139280 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-home" pod "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" (UID: "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:42.139318 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139298 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ww8k6\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.139449 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139419 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnf8\" (UniqueName: \"kubernetes.io/projected/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-kube-api-access-7cnf8\") pod \"maas-api-revoke-sub-3d4156f1-ww8k6\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.139540 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139517 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-model-cache" pod "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" (UID: "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:42.139598 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139559 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ww8k6\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.139700 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139684 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:42.139755 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139710 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:42.139755 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.139727 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:42.141192 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.141165 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tls-certs" pod "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" (UID: "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:42.141314 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.141208 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-kube-api-access-jmn9l" pod "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" (UID: "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe"). InnerVolumeSpecName "kube-api-access-jmn9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:42.141314 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.141265 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-dshm" pod "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" (UID: "bb310e4f-c5e9-41d9-abee-b3e2ececb2fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:42.240206 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.240179 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ww8k6\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.240348 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.240227 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnf8\" (UniqueName: \"kubernetes.io/projected/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-kube-api-access-7cnf8\") pod \"maas-api-revoke-sub-3d4156f1-ww8k6\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.240348 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.240276 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ww8k6\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.240348 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.240320 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmn9l\" (UniqueName: \"kubernetes.io/projected/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-kube-api-access-jmn9l\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:42.240348 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.240331 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:42.240348 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.240339 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:42.240914 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.240897 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ww8k6\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.242369 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.242349 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ww8k6\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.243302 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.243280 6287 generic.go:356] "Generic (PLEG): container finished" podID="bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" containerID="d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56" exitCode=0 Nov 01 23:02:42.243387 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.243349 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" Nov 01 23:02:42.243387 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.243369 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" event={"ID":"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe","Type":"ContainerDied","Data":"d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56"} Nov 01 23:02:42.243473 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.243410 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" event={"ID":"bb310e4f-c5e9-41d9-abee-b3e2ececb2fe","Type":"ContainerDied","Data":"6445f6c7bf0914c39d87a977963b19c7dde3954562f192ef10d23a887143781c"} Nov 01 23:02:42.243473 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.243427 6287 scope.go:122] "RemoveContainer" containerID="d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56" Nov 01 23:02:42.248673 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.248652 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnf8\" (UniqueName: \"kubernetes.io/projected/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-kube-api-access-7cnf8\") pod \"maas-api-revoke-sub-3d4156f1-ww8k6\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.252457 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.252442 6287 scope.go:122] "RemoveContainer" containerID="d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56" Nov 01 23:02:42.252732 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:02:42.252705 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56\": container with ID starting with d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56 not found: ID does not exist" containerID="d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56" Nov 01 23:02:42.252822 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.252737 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56"} err="failed to get container status \"d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56\": rpc error: code = NotFound desc = could not find container \"d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56\": container with ID starting with d8b08535e2b535e12637497d3544fd4805bd205644034c02e3a79bc5aad05e56 not found: ID does not exist" Nov 01 23:02:42.265344 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.265324 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v"] Nov 01 23:02:42.269955 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.269935 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v"] Nov 01 23:02:42.389275 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.389254 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:42.666314 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.666289 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6"] Nov 01 23:02:42.668796 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:02:42.668767 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5b24c1_9760_4d1e_9766_b6ddc1a369a5.slice/crio-7612c8403aa76ff0388008ccfa1e38289c3b6e653c52e79e298ddf3c6b4cdc7b WatchSource:0}: Error finding container 7612c8403aa76ff0388008ccfa1e38289c3b6e653c52e79e298ddf3c6b4cdc7b: Status 404 returned error can't find the container with id 7612c8403aa76ff0388008ccfa1e38289c3b6e653c52e79e298ddf3c6b4cdc7b Nov 01 23:02:42.953970 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:42.953901 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-7ae63f/simulator-single-main-7ae63f-kserve-6ddb6696b4-hhz5v" podUID="bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" containerName="main" probeResult="failure" output="Get \"https://10.133.1.2:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:02:43.251361 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:43.251325 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" event={"ID":"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5","Type":"ContainerStarted","Data":"890f371b1e503133cb442306aac7569e4d80154c9a8f6d2326d239b6f4c77377"} Nov 01 23:02:43.251554 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:43.251374 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" event={"ID":"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5","Type":"ContainerStarted","Data":"7612c8403aa76ff0388008ccfa1e38289c3b6e653c52e79e298ddf3c6b4cdc7b"} Nov 01 23:02:43.269627 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:43.269584 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" podStartSLOduration=1.269571392 podStartE2EDuration="1.269571392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:43.267723759 +0000 UTC m=+2375.916222995" watchObservedRunningTime="2026-11-01 23:02:43.269571392 +0000 UTC m=+2375.918070627" Nov 01 23:02:43.468894 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:43.468858 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0767a8ec-0052-4f5a-8662-2253e3178346" path="/var/lib/kubelet/pods/0767a8ec-0052-4f5a-8662-2253e3178346/volumes" Nov 01 23:02:43.469284 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:43.469270 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb310e4f-c5e9-41d9-abee-b3e2ececb2fe" path="/var/lib/kubelet/pods/bb310e4f-c5e9-41d9-abee-b3e2ececb2fe/volumes" Nov 01 23:02:44.257735 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:44.257699 6287 generic.go:356] "Generic (PLEG): container finished" podID="9d5b24c1-9760-4d1e-9766-b6ddc1a369a5" containerID="890f371b1e503133cb442306aac7569e4d80154c9a8f6d2326d239b6f4c77377" exitCode=0 Nov 01 23:02:44.258170 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:44.257782 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" event={"ID":"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5","Type":"ContainerDied","Data":"890f371b1e503133cb442306aac7569e4d80154c9a8f6d2326d239b6f4c77377"} Nov 01 23:02:45.393904 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.393878 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:45.470659 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.470625 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-kube-api-access-7cnf8\" (UniqueName: \"kubernetes.io/projected/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-kube-api-access-7cnf8\") pod \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " Nov 01 23:02:45.470864 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.470709 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-openshift-ca\") pod \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " Nov 01 23:02:45.470864 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.470741 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-xks-ca\") pod \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\" (UID: \"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5\") " Nov 01 23:02:45.471088 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.471065 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-openshift-ca" pod "9d5b24c1-9760-4d1e-9766-b6ddc1a369a5" (UID: "9d5b24c1-9760-4d1e-9766-b6ddc1a369a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:45.472766 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.472746 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-xks-ca" pod "9d5b24c1-9760-4d1e-9766-b6ddc1a369a5" (UID: "9d5b24c1-9760-4d1e-9766-b6ddc1a369a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:45.472863 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.472846 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-kube-api-access-7cnf8" pod "9d5b24c1-9760-4d1e-9766-b6ddc1a369a5" (UID: "9d5b24c1-9760-4d1e-9766-b6ddc1a369a5"). InnerVolumeSpecName "kube-api-access-7cnf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:45.571919 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.571831 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:45.571919 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.571864 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:45.571919 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:45.571878 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cnf8\" (UniqueName: \"kubernetes.io/projected/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5-kube-api-access-7cnf8\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:45.595355 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:02:45.595322 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:02:46.268536 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:46.268508 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" Nov 01 23:02:46.268718 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:46.268508 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6" event={"ID":"9d5b24c1-9760-4d1e-9766-b6ddc1a369a5","Type":"ContainerDied","Data":"7612c8403aa76ff0388008ccfa1e38289c3b6e653c52e79e298ddf3c6b4cdc7b"} Nov 01 23:02:46.268718 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:46.268632 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7612c8403aa76ff0388008ccfa1e38289c3b6e653c52e79e298ddf3c6b4cdc7b" Nov 01 23:02:47.824378 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:47.824335 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" podUID="40b23808-f190-4ecd-bfca-c0e1fba243b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:02:47.824378 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:47.824370 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" podUID="8e8604a4-2f6e-4212-84b0-6320b25ebca7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:02:57.280529 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.280467 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:02:57.315722 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.315699 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:02:57.332920 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.332867 6287 generic.go:356] "Generic (PLEG): container finished" podID="40b23808-f190-4ecd-bfca-c0e1fba243b3" containerID="373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa" exitCode=0 Nov 01 23:02:57.333170 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.332952 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" Nov 01 23:02:57.333170 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.333039 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" event={"ID":"40b23808-f190-4ecd-bfca-c0e1fba243b3","Type":"ContainerDied","Data":"373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa"} Nov 01 23:02:57.333408 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.333204 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m" event={"ID":"40b23808-f190-4ecd-bfca-c0e1fba243b3","Type":"ContainerDied","Data":"fcf0070694438b2d7b7d5dfec291d4100ea0b7a3b2065476c09387635608c8a9"} Nov 01 23:02:57.333408 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.333268 6287 scope.go:122] "RemoveContainer" containerID="373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa" Nov 01 23:02:57.338090 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.338063 6287 generic.go:356] "Generic (PLEG): container finished" podID="8e8604a4-2f6e-4212-84b0-6320b25ebca7" containerID="4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb" exitCode=0 Nov 01 23:02:57.338227 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.338206 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" event={"ID":"8e8604a4-2f6e-4212-84b0-6320b25ebca7","Type":"ContainerDied","Data":"4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb"} Nov 01 23:02:57.338302 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.338253 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" event={"ID":"8e8604a4-2f6e-4212-84b0-6320b25ebca7","Type":"ContainerDied","Data":"f352d9d067cfe7bb7fce1bcfae50d65af50b4bc655725afc9b3b84f26f8229d1"} Nov 01 23:02:57.338363 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.338325 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c" Nov 01 23:02:57.345188 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.345164 6287 scope.go:122] "RemoveContainer" containerID="373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa" Nov 01 23:02:57.345464 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:02:57.345444 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa\": container with ID starting with 373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa not found: ID does not exist" containerID="373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa" Nov 01 23:02:57.345547 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.345477 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa"} err="failed to get container status \"373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa\": rpc error: code = NotFound desc = could not find container \"373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa\": container with ID starting with 373a41ddb472d366348a09fee5c24aac61ef04dfc0e28f0d57075cdb7756e2fa not found: ID does not exist" Nov 01 23:02:57.345547 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.345503 6287 scope.go:122] "RemoveContainer" containerID="4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb" Nov 01 23:02:57.353875 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.353847 6287 scope.go:122] "RemoveContainer" containerID="4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb" Nov 01 23:02:57.354250 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:02:57.354224 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb\": container with ID starting with 4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb not found: ID does not exist" containerID="4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb" Nov 01 23:02:57.354328 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.354260 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb"} err="failed to get container status \"4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb\": rpc error: code = NotFound desc = could not find container \"4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb\": container with ID starting with 4c877d426f50e8df475da84a967291cb482ec876efab2dd121b8c32b7789d8eb not found: ID does not exist" Nov 01 23:02:57.380650 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.380623 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b23808-f190-4ecd-bfca-c0e1fba243b3-kube-api-access-z2vpw\" (UniqueName: \"kubernetes.io/projected/40b23808-f190-4ecd-bfca-c0e1fba243b3-kube-api-access-z2vpw\") pod \"40b23808-f190-4ecd-bfca-c0e1fba243b3\" (UID: \"40b23808-f190-4ecd-bfca-c0e1fba243b3\") " Nov 01 23:02:57.380827 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.380711 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8604a4-2f6e-4212-84b0-6320b25ebca7-kube-api-access-jhm4s\" (UniqueName: \"kubernetes.io/projected/8e8604a4-2f6e-4212-84b0-6320b25ebca7-kube-api-access-jhm4s\") pod \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\" (UID: \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\") " Nov 01 23:02:57.380827 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.380746 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b23808-f190-4ecd-bfca-c0e1fba243b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b23808-f190-4ecd-bfca-c0e1fba243b3-plugins-config-volume\") pod \"40b23808-f190-4ecd-bfca-c0e1fba243b3\" (UID: \"40b23808-f190-4ecd-bfca-c0e1fba243b3\") " Nov 01 23:02:57.380827 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.380766 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8604a4-2f6e-4212-84b0-6320b25ebca7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8604a4-2f6e-4212-84b0-6320b25ebca7-plugins-config-volume\") pod \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\" (UID: \"8e8604a4-2f6e-4212-84b0-6320b25ebca7\") " Nov 01 23:02:57.381157 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.381127 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b23808-f190-4ecd-bfca-c0e1fba243b3-plugins-config-volume" pod "40b23808-f190-4ecd-bfca-c0e1fba243b3" (UID: "40b23808-f190-4ecd-bfca-c0e1fba243b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:57.381268 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.381175 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8604a4-2f6e-4212-84b0-6320b25ebca7-plugins-config-volume" pod "8e8604a4-2f6e-4212-84b0-6320b25ebca7" (UID: "8e8604a4-2f6e-4212-84b0-6320b25ebca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:57.382810 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.382790 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b23808-f190-4ecd-bfca-c0e1fba243b3-kube-api-access-z2vpw" pod "40b23808-f190-4ecd-bfca-c0e1fba243b3" (UID: "40b23808-f190-4ecd-bfca-c0e1fba243b3"). InnerVolumeSpecName "kube-api-access-z2vpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:57.382893 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.382875 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8604a4-2f6e-4212-84b0-6320b25ebca7-kube-api-access-jhm4s" pod "8e8604a4-2f6e-4212-84b0-6320b25ebca7" (UID: "8e8604a4-2f6e-4212-84b0-6320b25ebca7"). InnerVolumeSpecName "kube-api-access-jhm4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:57.481842 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.481804 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2vpw\" (UniqueName: \"kubernetes.io/projected/40b23808-f190-4ecd-bfca-c0e1fba243b3-kube-api-access-z2vpw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:57.481842 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.481844 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhm4s\" (UniqueName: \"kubernetes.io/projected/8e8604a4-2f6e-4212-84b0-6320b25ebca7-kube-api-access-jhm4s\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:57.482085 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.481867 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b23808-f190-4ecd-bfca-c0e1fba243b3-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:57.482085 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.481882 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8604a4-2f6e-4212-84b0-6320b25ebca7-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:57.649442 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.649362 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m"] Nov 01 23:02:57.656421 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.656390 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ae63f-6b6dd9f8c8ln6m"] Nov 01 23:02:57.667712 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.667684 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c"] Nov 01 23:02:57.675842 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:57.675816 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ae63f-559f8mhn4c"] Nov 01 23:02:59.469281 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:59.469247 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b23808-f190-4ecd-bfca-c0e1fba243b3" path="/var/lib/kubelet/pods/40b23808-f190-4ecd-bfca-c0e1fba243b3/volumes" Nov 01 23:02:59.469662 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:02:59.469626 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8604a4-2f6e-4212-84b0-6320b25ebca7" path="/var/lib/kubelet/pods/8e8604a4-2f6e-4212-84b0-6320b25ebca7/volumes" Nov 01 23:03:10.568863 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.568830 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx"] Nov 01 23:03:10.569376 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.569310 6287 cpu_manager.go:414] "containerMap: removing container" podUID="40b23808-f190-4ecd-bfca-c0e1fba243b3" containerName="payload-processing" Nov 01 23:03:10.569376 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.569324 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b23808-f190-4ecd-bfca-c0e1fba243b3" containerName="payload-processing" Nov 01 23:03:10.569376 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.569338 6287 cpu_manager.go:414] "containerMap: removing container" podUID="8e8604a4-2f6e-4212-84b0-6320b25ebca7" containerName="payload-pre-processing" Nov 01 23:03:10.569376 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.569343 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8604a4-2f6e-4212-84b0-6320b25ebca7" containerName="payload-pre-processing" Nov 01 23:03:10.569376 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.569365 6287 cpu_manager.go:414] "containerMap: removing container" podUID="9d5b24c1-9760-4d1e-9766-b6ddc1a369a5" containerName="revoke-keys" Nov 01 23:03:10.569376 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.569370 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5b24c1-9760-4d1e-9766-b6ddc1a369a5" containerName="revoke-keys" Nov 01 23:03:10.569613 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.569430 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5b24c1-9760-4d1e-9766-b6ddc1a369a5" containerName="revoke-keys" Nov 01 23:03:10.569613 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.569441 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b23808-f190-4ecd-bfca-c0e1fba243b3" containerName="payload-processing" Nov 01 23:03:10.569613 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.569448 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8604a4-2f6e-4212-84b0-6320b25ebca7" containerName="payload-pre-processing" Nov 01 23:03:10.572123 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.572093 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.593938 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.593893 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.594148 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.593942 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.594148 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.593986 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.594148 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.594030 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.594148 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.594097 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.594391 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.594177 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d90fde-9034-4fa7-be99-57c1dfc881fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.594391 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.594308 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.594498 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.594397 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft8m7\" (UniqueName: \"kubernetes.io/projected/08d90fde-9034-4fa7-be99-57c1dfc881fd-kube-api-access-ft8m7\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.594498 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.594434 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.597902 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.597877 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx"] Nov 01 23:03:10.610883 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.610858 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9"] Nov 01 23:03:10.614967 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.614947 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.649902 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.649877 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9"] Nov 01 23:03:10.695832 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.695803 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.695991 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.695845 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.695991 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.695863 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.695991 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.695879 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d90fde-9034-4fa7-be99-57c1dfc881fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.696217 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696055 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.696217 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696085 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.696217 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696134 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.696401 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696270 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4769\" (UniqueName: \"kubernetes.io/projected/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-kube-api-access-m4769\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.696401 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696294 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.696401 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696346 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.696401 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696357 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8m7\" (UniqueName: \"kubernetes.io/projected/08d90fde-9034-4fa7-be99-57c1dfc881fd-kube-api-access-ft8m7\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.696401 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696398 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.696675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696429 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.696675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696465 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.696675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696534 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.696675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696559 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.696675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696589 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.696675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696592 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d90fde-9034-4fa7-be99-57c1dfc881fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.696675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696661 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.697083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696702 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.697083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696850 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.697083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.696886 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.697083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.697039 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.698366 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.698348 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.698609 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.698590 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.717022 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.716993 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d90fde-9034-4fa7-be99-57c1dfc881fd-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.717596 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.717574 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft8m7\" (UniqueName: \"kubernetes.io/projected/08d90fde-9034-4fa7-be99-57c1dfc881fd-kube-api-access-ft8m7\") pod \"maas-default-gateway-openshift-default-58847f9fc5-ncvpx\" (UID: \"08d90fde-9034-4fa7-be99-57c1dfc881fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.797746 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.797717 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.797922 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.797753 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.797922 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.797771 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.797922 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.797808 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.797922 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.797845 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.797922 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.797881 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.798145 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.797953 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.798145 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.797987 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.798145 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.798018 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4769\" (UniqueName: \"kubernetes.io/projected/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-kube-api-access-m4769\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.798256 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.798238 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.798316 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.798291 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.798404 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.798379 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.798506 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.798487 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.798599 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.798578 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.800508 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.800483 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.800668 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.800654 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.809224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.809200 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.809465 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.809444 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4769\" (UniqueName: \"kubernetes.io/projected/1f464e54-af08-4804-a3a2-a4ff4ac3ef0d-kube-api-access-m4769\") pod \"maas-default-gateway-openshift-default-58847f9fc5-fhcl9\" (UID: \"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:10.885319 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.885231 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:10.932286 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:10.932252 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:11.163409 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.163386 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx"] Nov 01 23:03:11.165792 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:03:11.165762 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d90fde_9034_4fa7_be99_57c1dfc881fd.slice/crio-c3a1a3986e35f3a959710fae81413ce429a828924c573d3a01357b30b3a292a9 WatchSource:0}: Error finding container c3a1a3986e35f3a959710fae81413ce429a828924c573d3a01357b30b3a292a9: Status 404 returned error can't find the container with id c3a1a3986e35f3a959710fae81413ce429a828924c573d3a01357b30b3a292a9 Nov 01 23:03:11.167992 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.167960 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:03:11.168065 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.168024 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:03:11.168065 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.168051 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:03:11.218899 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.218876 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9"] Nov 01 23:03:11.221332 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:03:11.221304 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f464e54_af08_4804_a3a2_a4ff4ac3ef0d.slice/crio-7433fd6c466a8760500d08b11448498de761b80dab2e4736013494aad8259ba5 WatchSource:0}: Error finding container 7433fd6c466a8760500d08b11448498de761b80dab2e4736013494aad8259ba5: Status 404 returned error can't find the container with id 7433fd6c466a8760500d08b11448498de761b80dab2e4736013494aad8259ba5 Nov 01 23:03:11.223773 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.223740 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:03:11.223848 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.223809 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:03:11.223848 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.223835 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:03:11.422520 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.422428 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" event={"ID":"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d","Type":"ContainerStarted","Data":"8223ff7369f41a6404020e6e0710c45c034b7e7b345d37a8edebcecc0d001b8d"} Nov 01 23:03:11.422520 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.422491 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" event={"ID":"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d","Type":"ContainerStarted","Data":"7433fd6c466a8760500d08b11448498de761b80dab2e4736013494aad8259ba5"} Nov 01 23:03:11.424412 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.424384 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" event={"ID":"08d90fde-9034-4fa7-be99-57c1dfc881fd","Type":"ContainerStarted","Data":"38779d29dea7c20471e5dcad781c321c6b52d74e76ff59b08fcbe9ebe2d71abe"} Nov 01 23:03:11.424557 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.424434 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" event={"ID":"08d90fde-9034-4fa7-be99-57c1dfc881fd","Type":"ContainerStarted","Data":"c3a1a3986e35f3a959710fae81413ce429a828924c573d3a01357b30b3a292a9"} Nov 01 23:03:11.444909 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.444857 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podStartSLOduration=1.444844347 podStartE2EDuration="1.444844347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:11.44204621 +0000 UTC m=+2404.090545448" watchObservedRunningTime="2026-11-01 23:03:11.444844347 +0000 UTC m=+2404.093343582" Nov 01 23:03:11.463039 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.462985 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podStartSLOduration=1.462967537 podStartE2EDuration="1.462967537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:11.461701655 +0000 UTC m=+2404.110200891" watchObservedRunningTime="2026-11-01 23:03:11.462967537 +0000 UTC m=+2404.111466774" Nov 01 23:03:11.885818 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.885780 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:11.933392 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:11.933358 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:12.891037 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:12.891008 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:12.937459 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:12.937434 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:13.434447 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:13.434413 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:13.434755 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:13.434476 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:13.435339 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:13.435317 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:03:13.435441 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:13.435368 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:03:15.586253 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:03:15.586217 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:03:17.346878 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:17.346848 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 23:03:17.347656 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:17.347639 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 23:03:19.774775 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:19.774747 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 23:03:19.775407 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:03:19.775390 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 23:03:45.589181 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:03:45.589140 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:04:15.587037 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:04:15.586989 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:04:34.151934 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.151899 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr"] Nov 01 23:04:34.155286 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.155269 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.164053 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.162355 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:04:34.164053 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.162409 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:04:34.164053 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.162500 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:04:34.164053 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.162541 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:04:34.165701 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.165673 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr"] Nov 01 23:04:34.321850 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.321809 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wrpfr\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.322194 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.322168 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wrpfr\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.322394 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.322377 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5flq\" (UniqueName: \"kubernetes.io/projected/f9d8422c-12e6-48cf-956e-fa6337896689-kube-api-access-n5flq\") pod \"maas-api-revoke-sub-d576e68b-wrpfr\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.423837 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.423735 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5flq\" (UniqueName: \"kubernetes.io/projected/f9d8422c-12e6-48cf-956e-fa6337896689-kube-api-access-n5flq\") pod \"maas-api-revoke-sub-d576e68b-wrpfr\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.424007 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.423843 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wrpfr\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.424007 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.423873 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wrpfr\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.424701 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.424656 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wrpfr\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.426382 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.426361 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wrpfr\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.432715 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.432688 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5flq\" (UniqueName: \"kubernetes.io/projected/f9d8422c-12e6-48cf-956e-fa6337896689-kube-api-access-n5flq\") pod \"maas-api-revoke-sub-d576e68b-wrpfr\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.487333 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.487295 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:34.823169 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.800755 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr"] Nov 01 23:04:34.931309 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:34.931270 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" event={"ID":"f9d8422c-12e6-48cf-956e-fa6337896689","Type":"ContainerStarted","Data":"e16af5a3b4fb4d39965c4b4cc9a29423ea0e9b1da373edc182ce7fb86d3eecb5"} Nov 01 23:04:35.939342 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:35.938918 6287 generic.go:356] "Generic (PLEG): container finished" podID="f9d8422c-12e6-48cf-956e-fa6337896689" containerID="cec3dd78275ea63d1bf839fd379f8426ff345bdc25ddc2f3efa3bf32cc0329b0" exitCode=0 Nov 01 23:04:35.939342 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:35.939008 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" event={"ID":"f9d8422c-12e6-48cf-956e-fa6337896689","Type":"ContainerDied","Data":"cec3dd78275ea63d1bf839fd379f8426ff345bdc25ddc2f3efa3bf32cc0329b0"} Nov 01 23:04:37.086308 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.086285 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:37.148929 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.148893 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-openshift-ca\") pod \"f9d8422c-12e6-48cf-956e-fa6337896689\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " Nov 01 23:04:37.149167 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.148945 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-xks-ca\") pod \"f9d8422c-12e6-48cf-956e-fa6337896689\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " Nov 01 23:04:37.149167 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.149011 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d8422c-12e6-48cf-956e-fa6337896689-kube-api-access-n5flq\" (UniqueName: \"kubernetes.io/projected/f9d8422c-12e6-48cf-956e-fa6337896689-kube-api-access-n5flq\") pod \"f9d8422c-12e6-48cf-956e-fa6337896689\" (UID: \"f9d8422c-12e6-48cf-956e-fa6337896689\") " Nov 01 23:04:37.149303 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.149281 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-openshift-ca" pod "f9d8422c-12e6-48cf-956e-fa6337896689" (UID: "f9d8422c-12e6-48cf-956e-fa6337896689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:04:37.151238 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.151211 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d8422c-12e6-48cf-956e-fa6337896689-kube-api-access-n5flq" pod "f9d8422c-12e6-48cf-956e-fa6337896689" (UID: "f9d8422c-12e6-48cf-956e-fa6337896689"). InnerVolumeSpecName "kube-api-access-n5flq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:37.151238 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.151227 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-xks-ca" pod "f9d8422c-12e6-48cf-956e-fa6337896689" (UID: "f9d8422c-12e6-48cf-956e-fa6337896689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:04:37.250738 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.250704 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:37.250738 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.250741 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d8422c-12e6-48cf-956e-fa6337896689-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:37.251009 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.250758 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5flq\" (UniqueName: \"kubernetes.io/projected/f9d8422c-12e6-48cf-956e-fa6337896689-kube-api-access-n5flq\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:37.953556 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.953519 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" event={"ID":"f9d8422c-12e6-48cf-956e-fa6337896689","Type":"ContainerDied","Data":"e16af5a3b4fb4d39965c4b4cc9a29423ea0e9b1da373edc182ce7fb86d3eecb5"} Nov 01 23:04:37.953775 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.953571 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e16af5a3b4fb4d39965c4b4cc9a29423ea0e9b1da373edc182ce7fb86d3eecb5" Nov 01 23:04:37.953775 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:37.953627 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr" Nov 01 23:04:41.976272 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.976237 6287 generic.go:356] "Generic (PLEG): container finished" podID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerID="38779d29dea7c20471e5dcad781c321c6b52d74e76ff59b08fcbe9ebe2d71abe" exitCode=0 Nov 01 23:04:41.976778 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.976308 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" event={"ID":"08d90fde-9034-4fa7-be99-57c1dfc881fd","Type":"ContainerDied","Data":"38779d29dea7c20471e5dcad781c321c6b52d74e76ff59b08fcbe9ebe2d71abe"} Nov 01 23:04:41.976778 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.976712 6287 scope.go:122] "RemoveContainer" containerID="38779d29dea7c20471e5dcad781c321c6b52d74e76ff59b08fcbe9ebe2d71abe" Nov 01 23:04:41.978020 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.977970 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:04:41.978020 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.978014 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" event={"ID":"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d","Type":"ContainerDied","Data":"8223ff7369f41a6404020e6e0710c45c034b7e7b345d37a8edebcecc0d001b8d"} Nov 01 23:04:41.978190 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.978079 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:04:41.978190 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.977996 6287 generic.go:356] "Generic (PLEG): container finished" podID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerID="8223ff7369f41a6404020e6e0710c45c034b7e7b345d37a8edebcecc0d001b8d" exitCode=0 Nov 01 23:04:41.978190 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.978146 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:04:41.978391 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.978376 6287 scope.go:122] "RemoveContainer" containerID="8223ff7369f41a6404020e6e0710c45c034b7e7b345d37a8edebcecc0d001b8d" Nov 01 23:04:41.979368 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.979339 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:04:41.979498 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.979398 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:04:41.979498 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:41.979431 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:04:43.013770 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.013717 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" event={"ID":"08d90fde-9034-4fa7-be99-57c1dfc881fd","Type":"ContainerStarted","Data":"0f9ab95d418dd92de113b927836df97a08b8931bd8298e1a4b42b7a8fa567c9e"} Nov 01 23:04:43.021026 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.020979 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" event={"ID":"1f464e54-af08-4804-a3a2-a4ff4ac3ef0d","Type":"ContainerStarted","Data":"49f3c9ede46644d3c6ebb5b94f2711cff6733a23df9f94336644aa8f52d34aa3"} Nov 01 23:04:43.434400 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.434322 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:04:43.434647 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.434628 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:04:43.891136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.889620 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:04:43.891136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.889776 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 01 23:04:43.891136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.889833 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 01 23:04:43.933410 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.933371 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:04:43.934414 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.934395 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body= Nov 01 23:04:43.934544 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:43.934431 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" Nov 01 23:04:44.890138 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:44.887558 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 01 23:04:44.890138 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:44.887619 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 01 23:04:44.933402 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:44.933356 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body= Nov 01 23:04:44.933593 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:44.933414 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" Nov 01 23:04:45.641014 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:04:45.640958 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:04:45.887563 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:45.887520 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 01 23:04:45.887910 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:45.887869 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 01 23:04:45.933152 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:45.933054 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body= Nov 01 23:04:45.933152 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:45.933105 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" Nov 01 23:04:46.887151 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:46.886632 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 01 23:04:46.887151 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:46.886714 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 01 23:04:46.933404 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:46.933368 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body= Nov 01 23:04:46.933808 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:46.933417 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" Nov 01 23:04:47.887562 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:47.887525 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 01 23:04:47.887870 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:47.887819 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 01 23:04:47.935682 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:47.935645 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body= Nov 01 23:04:47.936337 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:47.936290 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" Nov 01 23:04:48.886728 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:48.886673 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 01 23:04:48.887082 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:48.887033 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 01 23:04:48.936691 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:48.936652 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body= Nov 01 23:04:48.937343 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:48.937294 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" Nov 01 23:04:49.886953 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:49.886488 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Nov 01 23:04:49.886953 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:49.886549 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Nov 01 23:04:50.937596 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:50.937541 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:50.938152 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:50.937603 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:51.928347 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:51.928302 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:51.928586 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:51.928376 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:51.979334 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:51.979286 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:51.979845 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:51.979350 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:52.970321 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:52.970272 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:52.970499 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:52.970341 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:53.020374 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:53.020332 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:53.020918 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:53.020417 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:53.971131 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:53.971072 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:53.971358 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:53.971153 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:54.021257 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:54.021216 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:54.021757 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:54.021283 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:55.013389 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.013312 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-ncvpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:55.013619 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.013388 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" podUID="08d90fde-9034-4fa7-be99-57c1dfc881fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:55.065277 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.065229 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-fhcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:04:55.065685 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.065289 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" podUID="1f464e54-af08-4804-a3a2-a4ff4ac3ef0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:04:55.239906 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.239863 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7"] Nov 01 23:04:55.240490 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.240465 6287 cpu_manager.go:414] "containerMap: removing container" podUID="f9d8422c-12e6-48cf-956e-fa6337896689" containerName="revoke-keys" Nov 01 23:04:55.240689 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.240663 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d8422c-12e6-48cf-956e-fa6337896689" containerName="revoke-keys" Nov 01 23:04:55.240865 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.240836 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d8422c-12e6-48cf-956e-fa6337896689" containerName="revoke-keys" Nov 01 23:04:55.242999 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.242979 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7"] Nov 01 23:04:55.243125 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.243064 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.246356 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.246334 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:04:55.246505 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.246486 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:04:55.246645 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.246628 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:04:55.247361 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.247339 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:04:55.376505 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.376409 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gf9s7\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.376709 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.376677 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2nh\" (UniqueName: \"kubernetes.io/projected/48776640-c170-4b11-998f-bfc52c7b703f-kube-api-access-9t2nh\") pod \"maas-api-revoke-sub-d2bc93bc-gf9s7\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.376796 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.376739 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gf9s7\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.478398 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.478347 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gf9s7\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.478615 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.478433 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2nh\" (UniqueName: \"kubernetes.io/projected/48776640-c170-4b11-998f-bfc52c7b703f-kube-api-access-9t2nh\") pod \"maas-api-revoke-sub-d2bc93bc-gf9s7\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.478615 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.478473 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gf9s7\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.479266 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.479235 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gf9s7\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.481645 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.481615 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gf9s7\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.486967 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.486932 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2nh\" (UniqueName: \"kubernetes.io/projected/48776640-c170-4b11-998f-bfc52c7b703f-kube-api-access-9t2nh\") pod \"maas-api-revoke-sub-d2bc93bc-gf9s7\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.555844 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.555795 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:55.584202 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.584174 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:04:55.585407 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.585375 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-fhcl9" Nov 01 23:04:55.696998 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.696955 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:04:55.719635 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.719604 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-ncvpx" Nov 01 23:04:55.850221 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:55.850195 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7"] Nov 01 23:04:55.854278 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:04:55.854249 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48776640_c170_4b11_998f_bfc52c7b703f.slice/crio-2545fead453fbc7034c8bb0a227f0b4fe1ef03b9a87c822a61397edd69c3a0f4 WatchSource:0}: Error finding container 2545fead453fbc7034c8bb0a227f0b4fe1ef03b9a87c822a61397edd69c3a0f4: Status 404 returned error can't find the container with id 2545fead453fbc7034c8bb0a227f0b4fe1ef03b9a87c822a61397edd69c3a0f4 Nov 01 23:04:56.108277 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:56.108243 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" event={"ID":"48776640-c170-4b11-998f-bfc52c7b703f","Type":"ContainerStarted","Data":"2545fead453fbc7034c8bb0a227f0b4fe1ef03b9a87c822a61397edd69c3a0f4"} Nov 01 23:04:57.122747 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:57.122709 6287 generic.go:356] "Generic (PLEG): container finished" podID="48776640-c170-4b11-998f-bfc52c7b703f" containerID="f3e28cd7d7171b832d55d5e48d8cbd0d1fdacd0d17c560de3183608023227c2c" exitCode=0 Nov 01 23:04:57.123230 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:57.122801 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" event={"ID":"48776640-c170-4b11-998f-bfc52c7b703f","Type":"ContainerDied","Data":"f3e28cd7d7171b832d55d5e48d8cbd0d1fdacd0d17c560de3183608023227c2c"} Nov 01 23:04:58.261997 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.261973 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:58.404723 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.404641 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-openshift-ca\") pod \"48776640-c170-4b11-998f-bfc52c7b703f\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " Nov 01 23:04:58.404723 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.404680 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48776640-c170-4b11-998f-bfc52c7b703f-kube-api-access-9t2nh\" (UniqueName: \"kubernetes.io/projected/48776640-c170-4b11-998f-bfc52c7b703f-kube-api-access-9t2nh\") pod \"48776640-c170-4b11-998f-bfc52c7b703f\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " Nov 01 23:04:58.404961 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.404802 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-xks-ca\") pod \"48776640-c170-4b11-998f-bfc52c7b703f\" (UID: \"48776640-c170-4b11-998f-bfc52c7b703f\") " Nov 01 23:04:58.405080 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.405052 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-openshift-ca" pod "48776640-c170-4b11-998f-bfc52c7b703f" (UID: "48776640-c170-4b11-998f-bfc52c7b703f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:04:58.406831 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.406801 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-xks-ca" pod "48776640-c170-4b11-998f-bfc52c7b703f" (UID: "48776640-c170-4b11-998f-bfc52c7b703f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:04:58.406957 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.406863 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48776640-c170-4b11-998f-bfc52c7b703f-kube-api-access-9t2nh" pod "48776640-c170-4b11-998f-bfc52c7b703f" (UID: "48776640-c170-4b11-998f-bfc52c7b703f"). InnerVolumeSpecName "kube-api-access-9t2nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:58.506322 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.506291 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:58.506322 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.506318 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2nh\" (UniqueName: \"kubernetes.io/projected/48776640-c170-4b11-998f-bfc52c7b703f-kube-api-access-9t2nh\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:58.506500 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:58.506328 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48776640-c170-4b11-998f-bfc52c7b703f-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:59.134591 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:59.134563 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" Nov 01 23:04:59.134767 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:59.134557 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7" event={"ID":"48776640-c170-4b11-998f-bfc52c7b703f","Type":"ContainerDied","Data":"2545fead453fbc7034c8bb0a227f0b4fe1ef03b9a87c822a61397edd69c3a0f4"} Nov 01 23:04:59.134767 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:04:59.134697 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2545fead453fbc7034c8bb0a227f0b4fe1ef03b9a87c822a61397edd69c3a0f4" Nov 01 23:05:07.285667 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.285635 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr"] Nov 01 23:05:07.286089 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.286014 6287 cpu_manager.go:414] "containerMap: removing container" podUID="48776640-c170-4b11-998f-bfc52c7b703f" containerName="revoke-keys" Nov 01 23:05:07.286089 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.286024 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48776640-c170-4b11-998f-bfc52c7b703f" containerName="revoke-keys" Nov 01 23:05:07.286194 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.286092 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="48776640-c170-4b11-998f-bfc52c7b703f" containerName="revoke-keys" Nov 01 23:05:07.291888 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.291866 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.305013 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.304984 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ee0082-openshift-default-dockercfg-6pwrm\"" Nov 01 23:05:07.306939 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.306904 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr"] Nov 01 23:05:07.384532 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.384480 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wv44\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-kube-api-access-7wv44\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.384827 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.384801 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62555ef4-cb43-48b1-ba22-d4ded613db4c-istiod-ca-cert\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.385010 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.384993 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-credential-socket\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.385437 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.385376 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-token\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.385653 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.385629 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-data\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.385843 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.385816 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-certs\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.385997 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.385882 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-envoy\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.385997 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.385938 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-podinfo\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.386166 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.386028 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-socket\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487339 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487303 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-certs\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487539 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487349 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-envoy\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487539 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487367 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-podinfo\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487539 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487402 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-socket\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487539 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487433 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wv44\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-kube-api-access-7wv44\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487539 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487469 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62555ef4-cb43-48b1-ba22-d4ded613db4c-istiod-ca-cert\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487539 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487505 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-credential-socket\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487539 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487534 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-token\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487965 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487579 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-data\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487965 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487730 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-certs\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487965 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487779 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-socket\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.487965 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487934 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-credential-socket\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.488273 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.487981 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-data\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.488379 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.488358 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62555ef4-cb43-48b1-ba22-d4ded613db4c-istiod-ca-cert\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.489728 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.489706 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-podinfo\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.489728 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.489720 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-envoy\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.495444 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.495410 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-token\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.496189 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.496171 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wv44\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-kube-api-access-7wv44\") pod \"e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.605537 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.605453 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:07.888697 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.888669 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr"] Nov 01 23:05:07.891021 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:05:07.890988 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62555ef4_cb43_48b1_ba22_d4ded613db4c.slice/crio-93f4807fd12266f94f6268f33065705356bfd99bf0d5506f13152de36ff094dc WatchSource:0}: Error finding container 93f4807fd12266f94f6268f33065705356bfd99bf0d5506f13152de36ff094dc: Status 404 returned error can't find the container with id 93f4807fd12266f94f6268f33065705356bfd99bf0d5506f13152de36ff094dc Nov 01 23:05:07.893366 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.893329 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:05:07.893468 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.893397 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:05:07.893468 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:07.893430 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:05:08.183884 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:08.183798 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" event={"ID":"62555ef4-cb43-48b1-ba22-d4ded613db4c","Type":"ContainerStarted","Data":"ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380"} Nov 01 23:05:08.183884 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:08.183857 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" event={"ID":"62555ef4-cb43-48b1-ba22-d4ded613db4c","Type":"ContainerStarted","Data":"93f4807fd12266f94f6268f33065705356bfd99bf0d5506f13152de36ff094dc"} Nov 01 23:05:08.205869 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:08.205821 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" podStartSLOduration=1.205806688 podStartE2EDuration="1.205806688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:08.204756004 +0000 UTC m=+2520.853255238" watchObservedRunningTime="2026-11-01 23:05:08.205806688 +0000 UTC m=+2520.854305922" Nov 01 23:05:08.606087 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:08.606054 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:08.610967 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:08.610934 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:09.189278 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:09.189244 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:09.190393 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:09.190366 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:11.744741 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.744706 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz"] Nov 01 23:05:11.748155 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.748137 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:11.759315 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.759289 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a291003d-dockercfg-7vm7q\"" Nov 01 23:05:11.759763 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.759739 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz"] Nov 01 23:05:11.770495 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.770477 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ee0082\"" Nov 01 23:05:11.828537 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.828512 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a5b13-3226-49a7-b41e-6c76999ffb33-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz\" (UID: \"b02a5b13-3226-49a7-b41e-6c76999ffb33\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:11.828673 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.828559 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v52x2\" (UniqueName: \"kubernetes.io/projected/b02a5b13-3226-49a7-b41e-6c76999ffb33-kube-api-access-v52x2\") pod \"payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz\" (UID: \"b02a5b13-3226-49a7-b41e-6c76999ffb33\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:11.929689 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.929653 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a5b13-3226-49a7-b41e-6c76999ffb33-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz\" (UID: \"b02a5b13-3226-49a7-b41e-6c76999ffb33\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:11.929910 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.929699 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v52x2\" (UniqueName: \"kubernetes.io/projected/b02a5b13-3226-49a7-b41e-6c76999ffb33-kube-api-access-v52x2\") pod \"payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz\" (UID: \"b02a5b13-3226-49a7-b41e-6c76999ffb33\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:11.930428 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.930406 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a5b13-3226-49a7-b41e-6c76999ffb33-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz\" (UID: \"b02a5b13-3226-49a7-b41e-6c76999ffb33\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:11.938237 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:11.938216 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v52x2\" (UniqueName: \"kubernetes.io/projected/b02a5b13-3226-49a7-b41e-6c76999ffb33-kube-api-access-v52x2\") pod \"payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz\" (UID: \"b02a5b13-3226-49a7-b41e-6c76999ffb33\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:12.063253 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:12.063188 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:12.336586 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:12.336552 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz"] Nov 01 23:05:12.338957 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:05:12.338926 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02a5b13_3226_49a7_b41e_6c76999ffb33.slice/crio-ddb4bc65c891b9529b8dd3dfa0023faeade1e06677e778e06f63ddd565efe4ed WatchSource:0}: Error finding container ddb4bc65c891b9529b8dd3dfa0023faeade1e06677e778e06f63ddd565efe4ed: Status 404 returned error can't find the container with id ddb4bc65c891b9529b8dd3dfa0023faeade1e06677e778e06f63ddd565efe4ed Nov 01 23:05:13.210397 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:13.210357 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" event={"ID":"b02a5b13-3226-49a7-b41e-6c76999ffb33","Type":"ContainerStarted","Data":"9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04"} Nov 01 23:05:13.210802 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:13.210420 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" event={"ID":"b02a5b13-3226-49a7-b41e-6c76999ffb33","Type":"ContainerStarted","Data":"ddb4bc65c891b9529b8dd3dfa0023faeade1e06677e778e06f63ddd565efe4ed"} Nov 01 23:05:13.210802 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:13.210461 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:13.234603 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:13.234554 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" podStartSLOduration=2.234540949 podStartE2EDuration="2.234540949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:13.232405408 +0000 UTC m=+2525.880904644" watchObservedRunningTime="2026-11-01 23:05:13.234540949 +0000 UTC m=+2525.883040193" Nov 01 23:05:15.611568 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:05:15.611530 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:05:23.268587 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:23.268550 6287 generic.go:356] "Generic (PLEG): container finished" podID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerID="ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380" exitCode=0 Nov 01 23:05:23.269054 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:23.268623 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" event={"ID":"62555ef4-cb43-48b1-ba22-d4ded613db4c","Type":"ContainerDied","Data":"ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380"} Nov 01 23:05:23.269054 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:23.269000 6287 scope.go:122] "RemoveContainer" containerID="ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380" Nov 01 23:05:23.270198 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:23.270167 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:05:23.270295 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:23.270255 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:05:23.270342 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:23.270295 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:05:24.195138 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:24.194257 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:24.219332 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:24.219304 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:05:24.301198 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:24.301157 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" event={"ID":"62555ef4-cb43-48b1-ba22-d4ded613db4c","Type":"ContainerStarted","Data":"42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6"} Nov 01 23:05:24.605689 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:24.605653 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:24.607224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:24.607199 6287 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 23:05:24.607295 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:24.607252 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 23:05:25.606823 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:25.606783 6287 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 23:05:25.607332 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:25.606867 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 23:05:26.606180 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:26.606143 6287 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 23:05:26.606368 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:26.606198 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 23:05:27.122126 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.122081 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc"] Nov 01 23:05:27.126056 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.126029 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.130546 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.130520 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ee0082\"/\"simulator-single-main-ee0082-kserve-self-signed-certs\"" Nov 01 23:05:27.134406 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.134382 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ee0082\"/\"kube-root-ca.crt\"" Nov 01 23:05:27.138973 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.138954 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ee0082\"/\"openshift-service-ca.crt\"" Nov 01 23:05:27.142374 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.142351 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc"] Nov 01 23:05:27.195850 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.195815 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ee0082\"/\"default-dockercfg-dzlh9\"" Nov 01 23:05:27.228407 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.228374 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srk2w\" (UniqueName: \"kubernetes.io/projected/9619c55b-c92d-4881-ad0d-d92d4cc988a9-kube-api-access-srk2w\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.228585 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.228420 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-dshm\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.228585 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.228441 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tmp-dir\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.228585 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.228546 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-home\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.228585 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.228571 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-model-cache\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.228729 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.228602 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tls-certs\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.331394 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.331331 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srk2w\" (UniqueName: \"kubernetes.io/projected/9619c55b-c92d-4881-ad0d-d92d4cc988a9-kube-api-access-srk2w\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.331607 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.331422 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-dshm\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.331607 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.331447 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tmp-dir\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.331607 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.331504 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-home\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.331607 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.331532 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-model-cache\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.331607 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.331578 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tls-certs\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.332343 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.332316 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tmp-dir\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.333350 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.333327 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-home\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.333476 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.333459 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-model-cache\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.340737 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.340672 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-dshm\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.342438 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.342412 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srk2w\" (UniqueName: \"kubernetes.io/projected/9619c55b-c92d-4881-ad0d-d92d4cc988a9-kube-api-access-srk2w\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.343550 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.343521 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tls-certs\") pod \"simulator-single-main-ee0082-kserve-7c596d5994-k9skc\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.439041 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.438961 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:27.605983 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.605934 6287 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 23:05:27.606160 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.606005 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 23:05:27.741916 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:27.741887 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc"] Nov 01 23:05:27.744686 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:05:27.744654 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9619c55b_c92d_4881_ad0d_d92d4cc988a9.slice/crio-e734569ef5e372742da3c222c2e7be3cc0e2c63bf2f53e618ec434a6e80b4e7c WatchSource:0}: Error finding container e734569ef5e372742da3c222c2e7be3cc0e2c63bf2f53e618ec434a6e80b4e7c: Status 404 returned error can't find the container with id e734569ef5e372742da3c222c2e7be3cc0e2c63bf2f53e618ec434a6e80b4e7c Nov 01 23:05:28.324939 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:28.324898 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" event={"ID":"9619c55b-c92d-4881-ad0d-d92d4cc988a9","Type":"ContainerStarted","Data":"0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145"} Nov 01 23:05:28.324939 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:28.324949 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" event={"ID":"9619c55b-c92d-4881-ad0d-d92d4cc988a9","Type":"ContainerStarted","Data":"e734569ef5e372742da3c222c2e7be3cc0e2c63bf2f53e618ec434a6e80b4e7c"} Nov 01 23:05:28.355786 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:28.355735 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" podStartSLOduration=1.355723347 podStartE2EDuration="1.355723347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:28.352336612 +0000 UTC m=+2541.000835846" watchObservedRunningTime="2026-11-01 23:05:28.355723347 +0000 UTC m=+2541.004222582" Nov 01 23:05:28.439210 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:28.439171 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:28.606318 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:28.606230 6287 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Nov 01 23:05:28.606318 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:28.606282 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Nov 01 23:05:29.788807 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:29.788770 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:29.789440 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:29.789277 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:29.797644 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:29.797621 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:05:37.439754 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:37.439710 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:37.452255 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:37.452226 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:38.395701 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:38.395674 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:05:45.595771 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:05:45.595732 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:05:57.306885 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.306847 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd"] Nov 01 23:05:57.310809 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.310789 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.317919 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.317894 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:05:57.318067 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.318011 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:05:57.318067 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.318042 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:05:57.318208 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.318163 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:05:57.318538 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.318516 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd"] Nov 01 23:05:57.436257 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.436216 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-979beb82-2mrtd\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.436441 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.436265 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-979beb82-2mrtd\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.436441 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.436352 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pr6n\" (UniqueName: \"kubernetes.io/projected/2bd55e5c-c666-457a-8f5b-c53e7993683f-kube-api-access-5pr6n\") pod \"maas-api-revoke-sub-979beb82-2mrtd\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.537299 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.537250 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-979beb82-2mrtd\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.537497 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.537314 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pr6n\" (UniqueName: \"kubernetes.io/projected/2bd55e5c-c666-457a-8f5b-c53e7993683f-kube-api-access-5pr6n\") pod \"maas-api-revoke-sub-979beb82-2mrtd\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.537497 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.537474 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-979beb82-2mrtd\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.538083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.538065 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-979beb82-2mrtd\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.539644 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.539620 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-979beb82-2mrtd\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.545308 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.545285 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pr6n\" (UniqueName: \"kubernetes.io/projected/2bd55e5c-c666-457a-8f5b-c53e7993683f-kube-api-access-5pr6n\") pod \"maas-api-revoke-sub-979beb82-2mrtd\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.627722 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.627629 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:05:57.905587 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:57.905558 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd"] Nov 01 23:05:57.907473 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:05:57.907439 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd55e5c_c666_457a_8f5b_c53e7993683f.slice/crio-1a7f35cb0f419537a726d2844753b3d74f5b644303d464dfe9e9ee71d2c3ad9c WatchSource:0}: Error finding container 1a7f35cb0f419537a726d2844753b3d74f5b644303d464dfe9e9ee71d2c3ad9c: Status 404 returned error can't find the container with id 1a7f35cb0f419537a726d2844753b3d74f5b644303d464dfe9e9ee71d2c3ad9c Nov 01 23:05:58.503254 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:58.503220 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" event={"ID":"2bd55e5c-c666-457a-8f5b-c53e7993683f","Type":"ContainerStarted","Data":"d2d30b1b55145cb7931dbd904d03ce66212d166bf7c4389303f3edbbc44ecfb5"} Nov 01 23:05:58.503788 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:58.503272 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" event={"ID":"2bd55e5c-c666-457a-8f5b-c53e7993683f","Type":"ContainerStarted","Data":"1a7f35cb0f419537a726d2844753b3d74f5b644303d464dfe9e9ee71d2c3ad9c"} Nov 01 23:05:58.522664 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:58.522610 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" podStartSLOduration=1.522595855 podStartE2EDuration="1.522595855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:58.520348369 +0000 UTC m=+2571.168847604" watchObservedRunningTime="2026-11-01 23:05:58.522595855 +0000 UTC m=+2571.171095086" Nov 01 23:05:59.509486 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:59.509451 6287 generic.go:356] "Generic (PLEG): container finished" podID="2bd55e5c-c666-457a-8f5b-c53e7993683f" containerID="d2d30b1b55145cb7931dbd904d03ce66212d166bf7c4389303f3edbbc44ecfb5" exitCode=0 Nov 01 23:05:59.509894 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:05:59.509500 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" event={"ID":"2bd55e5c-c666-457a-8f5b-c53e7993683f","Type":"ContainerDied","Data":"d2d30b1b55145cb7931dbd904d03ce66212d166bf7c4389303f3edbbc44ecfb5"} Nov 01 23:06:00.651546 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.651522 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:06:00.767762 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.767670 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd55e5c-c666-457a-8f5b-c53e7993683f-kube-api-access-5pr6n\" (UniqueName: \"kubernetes.io/projected/2bd55e5c-c666-457a-8f5b-c53e7993683f-kube-api-access-5pr6n\") pod \"2bd55e5c-c666-457a-8f5b-c53e7993683f\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " Nov 01 23:06:00.767762 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.767726 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-xks-ca\") pod \"2bd55e5c-c666-457a-8f5b-c53e7993683f\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " Nov 01 23:06:00.767972 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.767801 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-openshift-ca\") pod \"2bd55e5c-c666-457a-8f5b-c53e7993683f\" (UID: \"2bd55e5c-c666-457a-8f5b-c53e7993683f\") " Nov 01 23:06:00.768223 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.768193 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-openshift-ca" pod "2bd55e5c-c666-457a-8f5b-c53e7993683f" (UID: "2bd55e5c-c666-457a-8f5b-c53e7993683f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:00.769890 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.769864 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd55e5c-c666-457a-8f5b-c53e7993683f-kube-api-access-5pr6n" pod "2bd55e5c-c666-457a-8f5b-c53e7993683f" (UID: "2bd55e5c-c666-457a-8f5b-c53e7993683f"). InnerVolumeSpecName "kube-api-access-5pr6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:00.769969 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.769863 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-xks-ca" pod "2bd55e5c-c666-457a-8f5b-c53e7993683f" (UID: "2bd55e5c-c666-457a-8f5b-c53e7993683f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:00.869494 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.869467 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:00.869494 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.869493 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pr6n\" (UniqueName: \"kubernetes.io/projected/2bd55e5c-c666-457a-8f5b-c53e7993683f-kube-api-access-5pr6n\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:00.869685 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:00.869507 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd55e5c-c666-457a-8f5b-c53e7993683f-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:01.522914 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:01.522880 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" Nov 01 23:06:01.522914 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:01.522911 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd" event={"ID":"2bd55e5c-c666-457a-8f5b-c53e7993683f","Type":"ContainerDied","Data":"1a7f35cb0f419537a726d2844753b3d74f5b644303d464dfe9e9ee71d2c3ad9c"} Nov 01 23:06:01.523164 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:01.522967 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a7f35cb0f419537a726d2844753b3d74f5b644303d464dfe9e9ee71d2c3ad9c" Nov 01 23:06:08.216762 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.216723 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk"] Nov 01 23:06:08.217330 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.217123 6287 cpu_manager.go:414] "containerMap: removing container" podUID="2bd55e5c-c666-457a-8f5b-c53e7993683f" containerName="revoke-keys" Nov 01 23:06:08.217330 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.217135 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd55e5c-c666-457a-8f5b-c53e7993683f" containerName="revoke-keys" Nov 01 23:06:08.217330 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.217240 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd55e5c-c666-457a-8f5b-c53e7993683f" containerName="revoke-keys" Nov 01 23:06:08.220231 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.220214 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.226365 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.226339 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:06:08.226542 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.226425 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:06:08.226648 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.226463 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:06:08.226750 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.226543 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:06:08.229807 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.229784 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk"] Nov 01 23:06:08.238832 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.238797 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.238980 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.238901 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.239063 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.238978 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlt4p\" (UniqueName: \"kubernetes.io/projected/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-kube-api-access-vlt4p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.340083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.340048 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlt4p\" (UniqueName: \"kubernetes.io/projected/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-kube-api-access-vlt4p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.340264 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.340138 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.340264 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.340195 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.340931 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.340901 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.342735 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.342713 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.348609 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.348587 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlt4p\" (UniqueName: \"kubernetes.io/projected/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-kube-api-access-vlt4p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.535591 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.535568 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:08.811206 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:08.811184 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk"] Nov 01 23:06:08.813253 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:06:08.813223 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b9ec8e_0350_42e1_b170_3f9abcd8b54c.slice/crio-d0faa027961f0e57cc7e35d229bd28568790bf7c670c8d78bb30e37c2431e5ac WatchSource:0}: Error finding container d0faa027961f0e57cc7e35d229bd28568790bf7c670c8d78bb30e37c2431e5ac: Status 404 returned error can't find the container with id d0faa027961f0e57cc7e35d229bd28568790bf7c670c8d78bb30e37c2431e5ac Nov 01 23:06:09.569162 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:09.569103 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" event={"ID":"30b9ec8e-0350-42e1-b170-3f9abcd8b54c","Type":"ContainerStarted","Data":"145d3297dc631fe677e531f0807dbc8da6a983b98edc54690ffb5be01e8ab85c"} Nov 01 23:06:09.569554 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:09.569169 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" event={"ID":"30b9ec8e-0350-42e1-b170-3f9abcd8b54c","Type":"ContainerStarted","Data":"d0faa027961f0e57cc7e35d229bd28568790bf7c670c8d78bb30e37c2431e5ac"} Nov 01 23:06:09.587419 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:09.587371 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" podStartSLOduration=1.587355501 podStartE2EDuration="1.587355501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:09.585510715 +0000 UTC m=+2582.234009950" watchObservedRunningTime="2026-11-01 23:06:09.587355501 +0000 UTC m=+2582.235854737" Nov 01 23:06:10.575738 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:10.575698 6287 generic.go:356] "Generic (PLEG): container finished" podID="30b9ec8e-0350-42e1-b170-3f9abcd8b54c" containerID="145d3297dc631fe677e531f0807dbc8da6a983b98edc54690ffb5be01e8ab85c" exitCode=0 Nov 01 23:06:10.576175 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:10.575793 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" event={"ID":"30b9ec8e-0350-42e1-b170-3f9abcd8b54c","Type":"ContainerDied","Data":"145d3297dc631fe677e531f0807dbc8da6a983b98edc54690ffb5be01e8ab85c"} Nov 01 23:06:11.726080 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.726053 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:11.775178 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.775136 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-openshift-ca\") pod \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " Nov 01 23:06:11.775375 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.775244 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-kube-api-access-vlt4p\" (UniqueName: \"kubernetes.io/projected/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-kube-api-access-vlt4p\") pod \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " Nov 01 23:06:11.775375 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.775272 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-xks-ca\") pod \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\" (UID: \"30b9ec8e-0350-42e1-b170-3f9abcd8b54c\") " Nov 01 23:06:11.775582 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.775553 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-openshift-ca" pod "30b9ec8e-0350-42e1-b170-3f9abcd8b54c" (UID: "30b9ec8e-0350-42e1-b170-3f9abcd8b54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:11.777371 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.777340 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-xks-ca" pod "30b9ec8e-0350-42e1-b170-3f9abcd8b54c" (UID: "30b9ec8e-0350-42e1-b170-3f9abcd8b54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:11.777488 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.777343 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-kube-api-access-vlt4p" pod "30b9ec8e-0350-42e1-b170-3f9abcd8b54c" (UID: "30b9ec8e-0350-42e1-b170-3f9abcd8b54c"). InnerVolumeSpecName "kube-api-access-vlt4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:11.877225 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.877152 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlt4p\" (UniqueName: \"kubernetes.io/projected/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-kube-api-access-vlt4p\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:11.877225 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.877179 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:11.877225 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:11.877192 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b9ec8e-0350-42e1-b170-3f9abcd8b54c-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:12.588507 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:12.588476 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" Nov 01 23:06:12.588690 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:12.588477 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk" event={"ID":"30b9ec8e-0350-42e1-b170-3f9abcd8b54c","Type":"ContainerDied","Data":"d0faa027961f0e57cc7e35d229bd28568790bf7c670c8d78bb30e37c2431e5ac"} Nov 01 23:06:12.588690 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:12.588599 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0faa027961f0e57cc7e35d229bd28568790bf7c670c8d78bb30e37c2431e5ac" Nov 01 23:06:13.079039 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:13.079002 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc"] Nov 01 23:06:15.587014 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:06:15.586972 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:06:18.335703 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:18.335663 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz"] Nov 01 23:06:19.179293 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:19.179256 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk"] Nov 01 23:06:19.182702 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:19.182673 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ee0082-wxzjk"] Nov 01 23:06:19.469087 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:19.469008 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b9ec8e-0350-42e1-b170-3f9abcd8b54c" path="/var/lib/kubelet/pods/30b9ec8e-0350-42e1-b170-3f9abcd8b54c/volumes" Nov 01 23:06:23.336273 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:23.336211 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" podUID="b02a5b13-3226-49a7-b41e-6c76999ffb33" containerName="payload-processing" containerID="cri-o://9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04" gracePeriod=25 Nov 01 23:06:24.217730 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:24.217690 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" podUID="b02a5b13-3226-49a7-b41e-6c76999ffb33" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:06:26.089527 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:26.089481 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr"] Nov 01 23:06:26.090010 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:26.089782 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" containerID="cri-o://42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6" gracePeriod=30 Nov 01 23:06:28.098799 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.098757 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" podUID="9619c55b-c92d-4881-ad0d-d92d4cc988a9" containerName="main" containerID="cri-o://0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145" gracePeriod=45 Nov 01 23:06:28.362819 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.362764 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:06:28.436843 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.436812 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-dshm\") pod \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " Nov 01 23:06:28.437012 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.436874 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-home\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-home\") pod \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " Nov 01 23:06:28.437012 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.436918 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tmp-dir\") pod \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " Nov 01 23:06:28.437012 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.436964 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tls-certs\") pod \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " Nov 01 23:06:28.437012 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.436998 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9619c55b-c92d-4881-ad0d-d92d4cc988a9-kube-api-access-srk2w\" (UniqueName: \"kubernetes.io/projected/9619c55b-c92d-4881-ad0d-d92d4cc988a9-kube-api-access-srk2w\") pod \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " Nov 01 23:06:28.437246 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.437025 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-model-cache\") pod \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\" (UID: \"9619c55b-c92d-4881-ad0d-d92d4cc988a9\") " Nov 01 23:06:28.437246 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.437195 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tmp-dir" pod "9619c55b-c92d-4881-ad0d-d92d4cc988a9" (UID: "9619c55b-c92d-4881-ad0d-d92d4cc988a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:28.437246 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.437207 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-home" pod "9619c55b-c92d-4881-ad0d-d92d4cc988a9" (UID: "9619c55b-c92d-4881-ad0d-d92d4cc988a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:28.437420 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.437356 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-model-cache" pod "9619c55b-c92d-4881-ad0d-d92d4cc988a9" (UID: "9619c55b-c92d-4881-ad0d-d92d4cc988a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:28.437462 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.437426 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:28.437462 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.437441 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:28.439067 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.439044 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-dshm" pod "9619c55b-c92d-4881-ad0d-d92d4cc988a9" (UID: "9619c55b-c92d-4881-ad0d-d92d4cc988a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:28.439314 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.439286 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tls-certs" pod "9619c55b-c92d-4881-ad0d-d92d4cc988a9" (UID: "9619c55b-c92d-4881-ad0d-d92d4cc988a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:28.439427 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.439359 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9619c55b-c92d-4881-ad0d-d92d4cc988a9-kube-api-access-srk2w" pod "9619c55b-c92d-4881-ad0d-d92d4cc988a9" (UID: "9619c55b-c92d-4881-ad0d-d92d4cc988a9"). InnerVolumeSpecName "kube-api-access-srk2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:28.537917 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.537886 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:28.537917 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.537914 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9619c55b-c92d-4881-ad0d-d92d4cc988a9-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:28.538148 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.537927 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srk2w\" (UniqueName: \"kubernetes.io/projected/9619c55b-c92d-4881-ad0d-d92d4cc988a9-kube-api-access-srk2w\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:28.538148 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.537937 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9619c55b-c92d-4881-ad0d-d92d4cc988a9-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:28.694005 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.693922 6287 generic.go:356] "Generic (PLEG): container finished" podID="9619c55b-c92d-4881-ad0d-d92d4cc988a9" containerID="0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145" exitCode=0 Nov 01 23:06:28.694005 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.693978 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" Nov 01 23:06:28.694249 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.694009 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" event={"ID":"9619c55b-c92d-4881-ad0d-d92d4cc988a9","Type":"ContainerDied","Data":"0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145"} Nov 01 23:06:28.694249 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.694062 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc" event={"ID":"9619c55b-c92d-4881-ad0d-d92d4cc988a9","Type":"ContainerDied","Data":"e734569ef5e372742da3c222c2e7be3cc0e2c63bf2f53e618ec434a6e80b4e7c"} Nov 01 23:06:28.694249 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.694089 6287 scope.go:122] "RemoveContainer" containerID="0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145" Nov 01 23:06:28.704446 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.704426 6287 scope.go:122] "RemoveContainer" containerID="0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145" Nov 01 23:06:28.704700 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:06:28.704680 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145\": container with ID starting with 0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145 not found: ID does not exist" containerID="0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145" Nov 01 23:06:28.704755 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.704718 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145"} err="failed to get container status \"0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145\": rpc error: code = NotFound desc = could not find container \"0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145\": container with ID starting with 0c85bc58d97ade43a6a1629da8179cce13db87e9608324c7bab7a78ce5fcb145 not found: ID does not exist" Nov 01 23:06:28.716884 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.716863 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc"] Nov 01 23:06:28.723090 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:28.723068 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ee0082/simulator-single-main-ee0082-kserve-7c596d5994-k9skc"] Nov 01 23:06:29.469409 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.469379 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9619c55b-c92d-4881-ad0d-d92d4cc988a9" path="/var/lib/kubelet/pods/9619c55b-c92d-4881-ad0d-d92d4cc988a9/volumes" Nov 01 23:06:29.621877 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.621831 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm"] Nov 01 23:06:29.622244 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.622227 6287 cpu_manager.go:414] "containerMap: removing container" podUID="9619c55b-c92d-4881-ad0d-d92d4cc988a9" containerName="main" Nov 01 23:06:29.622244 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.622242 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9619c55b-c92d-4881-ad0d-d92d4cc988a9" containerName="main" Nov 01 23:06:29.622466 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.622254 6287 cpu_manager.go:414] "containerMap: removing container" podUID="30b9ec8e-0350-42e1-b170-3f9abcd8b54c" containerName="revoke-keys" Nov 01 23:06:29.622466 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.622259 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b9ec8e-0350-42e1-b170-3f9abcd8b54c" containerName="revoke-keys" Nov 01 23:06:29.622466 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.622388 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9619c55b-c92d-4881-ad0d-d92d4cc988a9" containerName="main" Nov 01 23:06:29.622466 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.622395 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b9ec8e-0350-42e1-b170-3f9abcd8b54c" containerName="revoke-keys" Nov 01 23:06:29.626837 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.626811 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.635592 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.635560 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-cd542499-kserve-self-signed-certs\"" Nov 01 23:06:29.638582 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.638560 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm"] Nov 01 23:06:29.748592 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.748559 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdfe67f-638b-4494-8bf6-9a891badba9c-tls-certs\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.748803 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.748631 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-home\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.748803 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.748785 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-model-cache\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.748935 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.748844 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-tmp-dir\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.748935 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.748877 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2r7\" (UniqueName: \"kubernetes.io/projected/ebdfe67f-638b-4494-8bf6-9a891badba9c-kube-api-access-kf2r7\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.749052 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.748992 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-dshm\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.791472 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.791438 6287 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:06:29.791640 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.791501 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:06:29.850150 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.850092 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-home\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.850360 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.850167 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-model-cache\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.850360 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.850202 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-tmp-dir\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.850360 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.850233 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2r7\" (UniqueName: \"kubernetes.io/projected/ebdfe67f-638b-4494-8bf6-9a891badba9c-kube-api-access-kf2r7\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.850360 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.850259 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-dshm\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.850360 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.850331 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdfe67f-638b-4494-8bf6-9a891badba9c-tls-certs\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.851049 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.850625 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-home\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.851049 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.850738 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-model-cache\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.851049 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.850740 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-tmp-dir\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.852589 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.852569 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-dshm\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.852759 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.852742 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdfe67f-638b-4494-8bf6-9a891badba9c-tls-certs\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.858500 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.858475 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2r7\" (UniqueName: \"kubernetes.io/projected/ebdfe67f-638b-4494-8bf6-9a891badba9c-kube-api-access-kf2r7\") pod \"e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:29.940388 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:29.940364 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:30.214619 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:30.214588 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm"] Nov 01 23:06:30.216467 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:06:30.216440 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebdfe67f_638b_4494_8bf6_9a891badba9c.slice/crio-f53efe543002e991c231222a93275d0703e152fff38a928920b980b145272a70 WatchSource:0}: Error finding container f53efe543002e991c231222a93275d0703e152fff38a928920b980b145272a70: Status 404 returned error can't find the container with id f53efe543002e991c231222a93275d0703e152fff38a928920b980b145272a70 Nov 01 23:06:30.715955 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:30.715919 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" event={"ID":"ebdfe67f-638b-4494-8bf6-9a891badba9c","Type":"ContainerStarted","Data":"8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069"} Nov 01 23:06:30.716380 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:30.715964 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" event={"ID":"ebdfe67f-638b-4494-8bf6-9a891badba9c","Type":"ContainerStarted","Data":"f53efe543002e991c231222a93275d0703e152fff38a928920b980b145272a70"} Nov 01 23:06:30.732872 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:30.732826 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" podStartSLOduration=1.7328131679999998 podStartE2EDuration="1.732813168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:30.731916165 +0000 UTC m=+2603.380415402" watchObservedRunningTime="2026-11-01 23:06:30.732813168 +0000 UTC m=+2603.381312396" Nov 01 23:06:30.941168 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:30.941136 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:31.365418 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.365349 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:06:31.466267 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466229 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62555ef4-cb43-48b1-ba22-d4ded613db4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62555ef4-cb43-48b1-ba22-d4ded613db4c-istiod-ca-cert\") pod \"62555ef4-cb43-48b1-ba22-d4ded613db4c\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " Nov 01 23:06:31.466602 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466297 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-socket\") pod \"62555ef4-cb43-48b1-ba22-d4ded613db4c\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " Nov 01 23:06:31.466602 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466329 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-kube-api-access-7wv44\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-kube-api-access-7wv44\") pod \"62555ef4-cb43-48b1-ba22-d4ded613db4c\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " Nov 01 23:06:31.466602 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466364 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-token\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-token\") pod \"62555ef4-cb43-48b1-ba22-d4ded613db4c\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " Nov 01 23:06:31.466602 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466398 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-podinfo\") pod \"62555ef4-cb43-48b1-ba22-d4ded613db4c\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " Nov 01 23:06:31.466602 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466459 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-envoy\") pod \"62555ef4-cb43-48b1-ba22-d4ded613db4c\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " Nov 01 23:06:31.466602 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466536 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-credential-socket\") pod \"62555ef4-cb43-48b1-ba22-d4ded613db4c\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " Nov 01 23:06:31.466602 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466564 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-socket" pod "62555ef4-cb43-48b1-ba22-d4ded613db4c" (UID: "62555ef4-cb43-48b1-ba22-d4ded613db4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:31.466602 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466578 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-certs\") pod \"62555ef4-cb43-48b1-ba22-d4ded613db4c\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " Nov 01 23:06:31.467066 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466609 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-data\") pod \"62555ef4-cb43-48b1-ba22-d4ded613db4c\" (UID: \"62555ef4-cb43-48b1-ba22-d4ded613db4c\") " Nov 01 23:06:31.467066 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.466982 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.467238 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.467214 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-credential-socket" pod "62555ef4-cb43-48b1-ba22-d4ded613db4c" (UID: "62555ef4-cb43-48b1-ba22-d4ded613db4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:31.467310 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.467231 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-data" pod "62555ef4-cb43-48b1-ba22-d4ded613db4c" (UID: "62555ef4-cb43-48b1-ba22-d4ded613db4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:31.467399 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.467379 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-certs" pod "62555ef4-cb43-48b1-ba22-d4ded613db4c" (UID: "62555ef4-cb43-48b1-ba22-d4ded613db4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:31.467911 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.467882 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62555ef4-cb43-48b1-ba22-d4ded613db4c-istiod-ca-cert" pod "62555ef4-cb43-48b1-ba22-d4ded613db4c" (UID: "62555ef4-cb43-48b1-ba22-d4ded613db4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:31.470285 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.470255 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-envoy" pod "62555ef4-cb43-48b1-ba22-d4ded613db4c" (UID: "62555ef4-cb43-48b1-ba22-d4ded613db4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:31.470285 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.470260 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-podinfo" pod "62555ef4-cb43-48b1-ba22-d4ded613db4c" (UID: "62555ef4-cb43-48b1-ba22-d4ded613db4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:06:31.470453 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.470280 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-token" pod "62555ef4-cb43-48b1-ba22-d4ded613db4c" (UID: "62555ef4-cb43-48b1-ba22-d4ded613db4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:31.470453 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.470352 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-kube-api-access-7wv44" pod "62555ef4-cb43-48b1-ba22-d4ded613db4c" (UID: "62555ef4-cb43-48b1-ba22-d4ded613db4c"). InnerVolumeSpecName "kube-api-access-7wv44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:31.569156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.567825 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wv44\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-kube-api-access-7wv44\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.569156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.567867 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.569156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.567882 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.569156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.567896 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.569156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.567909 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.569156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.567929 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.569156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.567943 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62555ef4-cb43-48b1-ba22-d4ded613db4c-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.569156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.567956 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62555ef4-cb43-48b1-ba22-d4ded613db4c-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:31.727277 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.727183 6287 generic.go:356] "Generic (PLEG): container finished" podID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerID="42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6" exitCode=0 Nov 01 23:06:31.727837 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.727778 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" Nov 01 23:06:31.728010 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.727631 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" event={"ID":"62555ef4-cb43-48b1-ba22-d4ded613db4c","Type":"ContainerDied","Data":"42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6"} Nov 01 23:06:31.728091 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.728075 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr" event={"ID":"62555ef4-cb43-48b1-ba22-d4ded613db4c","Type":"ContainerDied","Data":"93f4807fd12266f94f6268f33065705356bfd99bf0d5506f13152de36ff094dc"} Nov 01 23:06:31.728174 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.728149 6287 scope.go:122] "RemoveContainer" containerID="42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6" Nov 01 23:06:31.750183 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.750153 6287 scope.go:122] "RemoveContainer" containerID="ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380" Nov 01 23:06:31.760168 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.755301 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr"] Nov 01 23:06:31.764874 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.764847 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee0082-openshift-default-66d5977cb9hdkdr"] Nov 01 23:06:31.772156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.772133 6287 scope.go:122] "RemoveContainer" containerID="42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6" Nov 01 23:06:31.772522 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:06:31.772490 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6\": container with ID starting with 42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6 not found: ID does not exist" containerID="42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6" Nov 01 23:06:31.772623 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.772529 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6"} err="failed to get container status \"42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6\": rpc error: code = NotFound desc = could not find container \"42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6\": container with ID starting with 42b5e5b3ea1c3403da7accba4c05979aaac9221e5666de20b9c2993a51aa1ed6 not found: ID does not exist" Nov 01 23:06:31.772623 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.772554 6287 scope.go:122] "RemoveContainer" containerID="ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380" Nov 01 23:06:31.772838 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:06:31.772815 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380\": container with ID starting with ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380 not found: ID does not exist" containerID="ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380" Nov 01 23:06:31.772950 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:31.772845 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380"} err="failed to get container status \"ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380\": rpc error: code = NotFound desc = could not find container \"ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380\": container with ID starting with ea41bd167248517bac625a4463f4b73b17d97dae4992da41472d7b326fbc9380 not found: ID does not exist" Nov 01 23:06:33.542901 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:33.542865 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" path="/var/lib/kubelet/pods/62555ef4-cb43-48b1-ba22-d4ded613db4c/volumes" Nov 01 23:06:34.231621 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:34.231211 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" podUID="b02a5b13-3226-49a7-b41e-6c76999ffb33" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:06:39.941245 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:39.941206 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:39.953577 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:39.953554 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:40.819048 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:40.819015 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:06:43.594383 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.594327 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:06:43.725131 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.725088 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02a5b13-3226-49a7-b41e-6c76999ffb33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a5b13-3226-49a7-b41e-6c76999ffb33-plugins-config-volume\") pod \"b02a5b13-3226-49a7-b41e-6c76999ffb33\" (UID: \"b02a5b13-3226-49a7-b41e-6c76999ffb33\") " Nov 01 23:06:43.725314 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.725239 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02a5b13-3226-49a7-b41e-6c76999ffb33-kube-api-access-v52x2\" (UniqueName: \"kubernetes.io/projected/b02a5b13-3226-49a7-b41e-6c76999ffb33-kube-api-access-v52x2\") pod \"b02a5b13-3226-49a7-b41e-6c76999ffb33\" (UID: \"b02a5b13-3226-49a7-b41e-6c76999ffb33\") " Nov 01 23:06:43.725514 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.725491 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02a5b13-3226-49a7-b41e-6c76999ffb33-plugins-config-volume" pod "b02a5b13-3226-49a7-b41e-6c76999ffb33" (UID: "b02a5b13-3226-49a7-b41e-6c76999ffb33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:43.727268 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.727249 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02a5b13-3226-49a7-b41e-6c76999ffb33-kube-api-access-v52x2" pod "b02a5b13-3226-49a7-b41e-6c76999ffb33" (UID: "b02a5b13-3226-49a7-b41e-6c76999ffb33"). InnerVolumeSpecName "kube-api-access-v52x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:43.825834 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.825805 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v52x2\" (UniqueName: \"kubernetes.io/projected/b02a5b13-3226-49a7-b41e-6c76999ffb33-kube-api-access-v52x2\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:43.825964 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.825838 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02a5b13-3226-49a7-b41e-6c76999ffb33-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:43.826490 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.826464 6287 generic.go:356] "Generic (PLEG): container finished" podID="b02a5b13-3226-49a7-b41e-6c76999ffb33" containerID="9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04" exitCode=0 Nov 01 23:06:43.826594 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.826510 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" Nov 01 23:06:43.826594 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.826516 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" event={"ID":"b02a5b13-3226-49a7-b41e-6c76999ffb33","Type":"ContainerDied","Data":"9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04"} Nov 01 23:06:43.826594 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.826571 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz" event={"ID":"b02a5b13-3226-49a7-b41e-6c76999ffb33","Type":"ContainerDied","Data":"ddb4bc65c891b9529b8dd3dfa0023faeade1e06677e778e06f63ddd565efe4ed"} Nov 01 23:06:43.826594 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.826594 6287 scope.go:122] "RemoveContainer" containerID="9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04" Nov 01 23:06:43.836251 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.836234 6287 scope.go:122] "RemoveContainer" containerID="9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04" Nov 01 23:06:43.836551 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:06:43.836533 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04\": container with ID starting with 9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04 not found: ID does not exist" containerID="9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04" Nov 01 23:06:43.836602 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.836560 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04"} err="failed to get container status \"9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04\": rpc error: code = NotFound desc = could not find container \"9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04\": container with ID starting with 9ecb4c11d8d01dbf44d9fd7666e3c5db4f6f142e10b97dfa47d5fbb3466a3b04 not found: ID does not exist" Nov 01 23:06:43.848124 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.848065 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz"] Nov 01 23:06:43.851900 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:43.851880 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee0082-76f85d99fsxfhz"] Nov 01 23:06:45.469523 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:45.469492 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02a5b13-3226-49a7-b41e-6c76999ffb33" path="/var/lib/kubelet/pods/b02a5b13-3226-49a7-b41e-6c76999ffb33/volumes" Nov 01 23:06:45.586320 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:06:45.586288 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:06:46.801306 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801208 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp"] Nov 01 23:06:46.801797 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801662 6287 cpu_manager.go:414] "containerMap: removing container" podUID="b02a5b13-3226-49a7-b41e-6c76999ffb33" containerName="payload-processing" Nov 01 23:06:46.801797 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801678 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02a5b13-3226-49a7-b41e-6c76999ffb33" containerName="payload-processing" Nov 01 23:06:46.801797 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801693 6287 cpu_manager.go:414] "containerMap: removing container" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" Nov 01 23:06:46.801797 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801701 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" Nov 01 23:06:46.801797 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801716 6287 cpu_manager.go:414] "containerMap: removing container" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" Nov 01 23:06:46.801797 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801721 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" Nov 01 23:06:46.802196 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801815 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02a5b13-3226-49a7-b41e-6c76999ffb33" containerName="payload-processing" Nov 01 23:06:46.802196 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801830 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" Nov 01 23:06:46.802196 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.801839 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="62555ef4-cb43-48b1-ba22-d4ded613db4c" containerName="istio-proxy" Nov 01 23:06:46.806695 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.806672 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:46.819236 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.819204 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp"] Nov 01 23:06:46.820215 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.820188 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-cd542499-kserve-self-signed-certs\"" Nov 01 23:06:46.954227 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.954185 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgwvk\" (UniqueName: \"kubernetes.io/projected/243e1b04-5d36-4252-921a-f83436903ede-kube-api-access-xgwvk\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:46.954409 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.954233 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-dshm\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:46.954409 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.954319 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243e1b04-5d36-4252-921a-f83436903ede-tls-certs\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:46.954493 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.954419 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-model-cache\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:46.954493 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.954438 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-home\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:46.954493 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:46.954454 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-tmp-dir\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.055181 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.055070 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwvk\" (UniqueName: \"kubernetes.io/projected/243e1b04-5d36-4252-921a-f83436903ede-kube-api-access-xgwvk\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.055181 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.055105 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-dshm\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.055181 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.055158 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243e1b04-5d36-4252-921a-f83436903ede-tls-certs\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.055486 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.055207 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-model-cache\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.055486 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.055344 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-home\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.055486 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.055382 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-tmp-dir\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.055667 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.055657 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-model-cache\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.055746 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.055722 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-home\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.055794 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.055750 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-tmp-dir\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.057362 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.057335 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-dshm\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.057639 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.057622 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243e1b04-5d36-4252-921a-f83436903ede-tls-certs\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.065177 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.065150 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwvk\" (UniqueName: \"kubernetes.io/projected/243e1b04-5d36-4252-921a-f83436903ede-kube-api-access-xgwvk\") pod \"e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.125054 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.125029 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:47.401516 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.401490 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp"] Nov 01 23:06:47.403292 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:06:47.403264 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod243e1b04_5d36_4252_921a_f83436903ede.slice/crio-d6f37ddf75fbddf472ec6915b88dc89a9537eb669fb302e619104401c17c96c6 WatchSource:0}: Error finding container d6f37ddf75fbddf472ec6915b88dc89a9537eb669fb302e619104401c17c96c6: Status 404 returned error can't find the container with id d6f37ddf75fbddf472ec6915b88dc89a9537eb669fb302e619104401c17c96c6 Nov 01 23:06:47.855235 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.855192 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" event={"ID":"243e1b04-5d36-4252-921a-f83436903ede","Type":"ContainerStarted","Data":"588994fbf0a33c78a8b762cff17368701b07484fbace2c1a44606e391c35b355"} Nov 01 23:06:47.855235 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.855236 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" event={"ID":"243e1b04-5d36-4252-921a-f83436903ede","Type":"ContainerStarted","Data":"d6f37ddf75fbddf472ec6915b88dc89a9537eb669fb302e619104401c17c96c6"} Nov 01 23:06:47.873575 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:47.873518 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" podStartSLOduration=1.8735041460000001 podStartE2EDuration="1.873504146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:47.87250135 +0000 UTC m=+2620.521000586" watchObservedRunningTime="2026-11-01 23:06:47.873504146 +0000 UTC m=+2620.522003381" Nov 01 23:06:48.125948 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:48.125871 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:57.125210 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:57.125169 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:57.137740 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:57.137705 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:06:57.986764 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:06:57.986731 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:07:14.446023 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:14.445986 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm"] Nov 01 23:07:15.020444 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:15.020410 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv"] Nov 01 23:07:15.025319 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:15.025288 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18448dea-q9jjv"] Nov 01 23:07:15.469499 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:15.469417 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455b0b80-8bf6-4709-8276-d58ec2fc26db" path="/var/lib/kubelet/pods/455b0b80-8bf6-4709-8276-d58ec2fc26db/volumes" Nov 01 23:07:15.586945 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:07:15.586905 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:07:16.115108 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:16.115069 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp"] Nov 01 23:07:18.597850 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:18.597809 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:07:18.598506 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:18.598468 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:07:29.466836 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.466794 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" podUID="ebdfe67f-638b-4494-8bf6-9a891badba9c" containerName="main" containerID="cri-o://8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069" gracePeriod=45 Nov 01 23:07:29.715147 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.715105 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:07:29.761831 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.761797 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-home\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-home\") pod \"ebdfe67f-638b-4494-8bf6-9a891badba9c\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " Nov 01 23:07:29.762028 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.761843 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-tmp-dir\") pod \"ebdfe67f-638b-4494-8bf6-9a891badba9c\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " Nov 01 23:07:29.762028 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.761888 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdfe67f-638b-4494-8bf6-9a891badba9c-kube-api-access-kf2r7\" (UniqueName: \"kubernetes.io/projected/ebdfe67f-638b-4494-8bf6-9a891badba9c-kube-api-access-kf2r7\") pod \"ebdfe67f-638b-4494-8bf6-9a891badba9c\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " Nov 01 23:07:29.762028 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.761940 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-dshm\") pod \"ebdfe67f-638b-4494-8bf6-9a891badba9c\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " Nov 01 23:07:29.762196 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.762055 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-home" pod "ebdfe67f-638b-4494-8bf6-9a891badba9c" (UID: "ebdfe67f-638b-4494-8bf6-9a891badba9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:29.762196 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.762108 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebdfe67f-638b-4494-8bf6-9a891badba9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdfe67f-638b-4494-8bf6-9a891badba9c-tls-certs\") pod \"ebdfe67f-638b-4494-8bf6-9a891badba9c\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " Nov 01 23:07:29.762196 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.762153 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-model-cache\") pod \"ebdfe67f-638b-4494-8bf6-9a891badba9c\" (UID: \"ebdfe67f-638b-4494-8bf6-9a891badba9c\") " Nov 01 23:07:29.762196 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.762122 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-tmp-dir" pod "ebdfe67f-638b-4494-8bf6-9a891badba9c" (UID: "ebdfe67f-638b-4494-8bf6-9a891badba9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:29.762454 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.762434 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:29.762517 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.762454 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-model-cache" pod "ebdfe67f-638b-4494-8bf6-9a891badba9c" (UID: "ebdfe67f-638b-4494-8bf6-9a891badba9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:29.762517 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.762458 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:29.764365 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.764331 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebdfe67f-638b-4494-8bf6-9a891badba9c-tls-certs" pod "ebdfe67f-638b-4494-8bf6-9a891badba9c" (UID: "ebdfe67f-638b-4494-8bf6-9a891badba9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:29.764365 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.764337 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-dshm" pod "ebdfe67f-638b-4494-8bf6-9a891badba9c" (UID: "ebdfe67f-638b-4494-8bf6-9a891badba9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:29.764563 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.764386 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdfe67f-638b-4494-8bf6-9a891badba9c-kube-api-access-kf2r7" pod "ebdfe67f-638b-4494-8bf6-9a891badba9c" (UID: "ebdfe67f-638b-4494-8bf6-9a891badba9c"). InnerVolumeSpecName "kube-api-access-kf2r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:29.863303 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.863257 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebdfe67f-638b-4494-8bf6-9a891badba9c-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:29.863303 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.863297 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:29.863303 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.863314 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2r7\" (UniqueName: \"kubernetes.io/projected/ebdfe67f-638b-4494-8bf6-9a891badba9c-kube-api-access-kf2r7\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:29.863552 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:29.863328 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebdfe67f-638b-4494-8bf6-9a891badba9c-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:30.298521 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:30.298486 6287 generic.go:356] "Generic (PLEG): container finished" podID="ebdfe67f-638b-4494-8bf6-9a891badba9c" containerID="8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069" exitCode=0 Nov 01 23:07:30.298704 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:30.298529 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" Nov 01 23:07:30.298704 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:30.298567 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" event={"ID":"ebdfe67f-638b-4494-8bf6-9a891badba9c","Type":"ContainerDied","Data":"8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069"} Nov 01 23:07:30.298704 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:30.298609 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm" event={"ID":"ebdfe67f-638b-4494-8bf6-9a891badba9c","Type":"ContainerDied","Data":"f53efe543002e991c231222a93275d0703e152fff38a928920b980b145272a70"} Nov 01 23:07:30.298704 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:30.298625 6287 scope.go:122] "RemoveContainer" containerID="8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069" Nov 01 23:07:30.307971 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:30.307955 6287 scope.go:122] "RemoveContainer" containerID="8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069" Nov 01 23:07:30.308260 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:07:30.308241 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069\": container with ID starting with 8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069 not found: ID does not exist" containerID="8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069" Nov 01 23:07:30.308355 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:30.308270 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069"} err="failed to get container status \"8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069\": rpc error: code = NotFound desc = could not find container \"8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069\": container with ID starting with 8e4cf011c57ffefcc946646ee58399fa0ce75dc27368b334401bfabf650bd069 not found: ID does not exist" Nov 01 23:07:30.320678 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:30.320657 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm"] Nov 01 23:07:30.324230 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:30.324209 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-cd542499-kserve-7968c55b97-lrgxm"] Nov 01 23:07:31.147136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:31.146411 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" podUID="243e1b04-5d36-4252-921a-f83436903ede" containerName="main" containerID="cri-o://588994fbf0a33c78a8b762cff17368701b07484fbace2c1a44606e391c35b355" gracePeriod=45 Nov 01 23:07:31.339443 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:31.339297 6287 generic.go:356] "Generic (PLEG): container finished" podID="243e1b04-5d36-4252-921a-f83436903ede" containerID="588994fbf0a33c78a8b762cff17368701b07484fbace2c1a44606e391c35b355" exitCode=0 Nov 01 23:07:31.339443 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:31.339378 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" event={"ID":"243e1b04-5d36-4252-921a-f83436903ede","Type":"ContainerDied","Data":"588994fbf0a33c78a8b762cff17368701b07484fbace2c1a44606e391c35b355"} Nov 01 23:07:31.490024 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:31.489990 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebdfe67f-638b-4494-8bf6-9a891badba9c" path="/var/lib/kubelet/pods/ebdfe67f-638b-4494-8bf6-9a891badba9c/volumes" Nov 01 23:07:31.932387 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:31.932347 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:07:32.035136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.034469 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-model-cache\") pod \"243e1b04-5d36-4252-921a-f83436903ede\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " Nov 01 23:07:32.035136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.034539 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-tmp-dir\") pod \"243e1b04-5d36-4252-921a-f83436903ede\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " Nov 01 23:07:32.035136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.034596 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-dshm\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-dshm\") pod \"243e1b04-5d36-4252-921a-f83436903ede\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " Nov 01 23:07:32.035136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.034627 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243e1b04-5d36-4252-921a-f83436903ede-kube-api-access-xgwvk\" (UniqueName: \"kubernetes.io/projected/243e1b04-5d36-4252-921a-f83436903ede-kube-api-access-xgwvk\") pod \"243e1b04-5d36-4252-921a-f83436903ede\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " Nov 01 23:07:32.035136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.034658 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-home\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-home\") pod \"243e1b04-5d36-4252-921a-f83436903ede\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " Nov 01 23:07:32.035136 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.034710 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243e1b04-5d36-4252-921a-f83436903ede-tls-certs\" (UniqueName: \"kubernetes.io/secret/243e1b04-5d36-4252-921a-f83436903ede-tls-certs\") pod \"243e1b04-5d36-4252-921a-f83436903ede\" (UID: \"243e1b04-5d36-4252-921a-f83436903ede\") " Nov 01 23:07:32.036444 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.036414 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-model-cache" pod "243e1b04-5d36-4252-921a-f83436903ede" (UID: "243e1b04-5d36-4252-921a-f83436903ede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:32.038219 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.038186 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-tmp-dir" pod "243e1b04-5d36-4252-921a-f83436903ede" (UID: "243e1b04-5d36-4252-921a-f83436903ede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:32.052423 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.052395 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-home" pod "243e1b04-5d36-4252-921a-f83436903ede" (UID: "243e1b04-5d36-4252-921a-f83436903ede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:32.057579 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.057539 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243e1b04-5d36-4252-921a-f83436903ede-kube-api-access-xgwvk" pod "243e1b04-5d36-4252-921a-f83436903ede" (UID: "243e1b04-5d36-4252-921a-f83436903ede"). InnerVolumeSpecName "kube-api-access-xgwvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:32.059243 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.059218 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243e1b04-5d36-4252-921a-f83436903ede-tls-certs" pod "243e1b04-5d36-4252-921a-f83436903ede" (UID: "243e1b04-5d36-4252-921a-f83436903ede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:32.059921 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.059900 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-dshm" pod "243e1b04-5d36-4252-921a-f83436903ede" (UID: "243e1b04-5d36-4252-921a-f83436903ede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:32.135898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.135710 6287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-tmp-dir\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:32.135898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.135756 6287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-dshm\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:32.135898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.135775 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgwvk\" (UniqueName: \"kubernetes.io/projected/243e1b04-5d36-4252-921a-f83436903ede-kube-api-access-xgwvk\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:32.135898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.135791 6287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-home\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:32.135898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.135804 6287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243e1b04-5d36-4252-921a-f83436903ede-tls-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:32.135898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.135818 6287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/243e1b04-5d36-4252-921a-f83436903ede-model-cache\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:32.300147 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.296692 6287 scope.go:122] "RemoveContainer" containerID="5e56a75affd3283e0447c3a2d18ea74d6bd6799aa46060b68281dd76cd4ac53b" Nov 01 23:07:32.408193 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.406248 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8m4jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:07:32.408193 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.406350 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8m4jm" podUID="e65c3aec-3fe4-4c4f-8689-cbabab25e810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:07:32.425142 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.421666 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" event={"ID":"243e1b04-5d36-4252-921a-f83436903ede","Type":"ContainerDied","Data":"d6f37ddf75fbddf472ec6915b88dc89a9537eb669fb302e619104401c17c96c6"} Nov 01 23:07:32.425142 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.421725 6287 scope.go:122] "RemoveContainer" containerID="588994fbf0a33c78a8b762cff17368701b07484fbace2c1a44606e391c35b355" Nov 01 23:07:32.425142 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.421876 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp" Nov 01 23:07:32.504954 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.502044 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp"] Nov 01 23:07:32.537150 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:32.523037 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-cd542499-kserve-7cdd755c69-gjszp"] Nov 01 23:07:33.472847 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:33.472798 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243e1b04-5d36-4252-921a-f83436903ede" path="/var/lib/kubelet/pods/243e1b04-5d36-4252-921a-f83436903ede/volumes" Nov 01 23:07:39.733236 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.733203 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb"] Nov 01 23:07:39.733633 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.733560 6287 cpu_manager.go:414] "containerMap: removing container" podUID="ebdfe67f-638b-4494-8bf6-9a891badba9c" containerName="main" Nov 01 23:07:39.733633 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.733571 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdfe67f-638b-4494-8bf6-9a891badba9c" containerName="main" Nov 01 23:07:39.733633 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.733590 6287 cpu_manager.go:414] "containerMap: removing container" podUID="243e1b04-5d36-4252-921a-f83436903ede" containerName="main" Nov 01 23:07:39.733633 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.733595 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243e1b04-5d36-4252-921a-f83436903ede" containerName="main" Nov 01 23:07:39.733808 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.733664 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdfe67f-638b-4494-8bf6-9a891badba9c" containerName="main" Nov 01 23:07:39.733808 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.733673 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="243e1b04-5d36-4252-921a-f83436903ede" containerName="main" Nov 01 23:07:39.736744 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.736727 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:39.744007 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.743966 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:07:39.744722 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.744698 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:07:39.748285 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.748261 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb"] Nov 01 23:07:39.751673 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.751649 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:07:39.751767 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.751718 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlj2w\"" Nov 01 23:07:39.860313 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.860280 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44e4d6c-fflqb\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:39.860313 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.860313 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5ggf\" (UniqueName: \"kubernetes.io/projected/e75d385f-258f-42e0-815b-375ba45f83f1-kube-api-access-b5ggf\") pod \"maas-api-revoke-sub-c44e4d6c-fflqb\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:39.860528 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.860387 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44e4d6c-fflqb\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:39.961178 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.961142 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44e4d6c-fflqb\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:39.961389 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.961279 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44e4d6c-fflqb\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:39.961389 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.961310 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5ggf\" (UniqueName: \"kubernetes.io/projected/e75d385f-258f-42e0-815b-375ba45f83f1-kube-api-access-b5ggf\") pod \"maas-api-revoke-sub-c44e4d6c-fflqb\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:39.961793 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.961769 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44e4d6c-fflqb\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:39.963562 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.963543 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44e4d6c-fflqb\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:39.972585 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:39.972558 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5ggf\" (UniqueName: \"kubernetes.io/projected/e75d385f-258f-42e0-815b-375ba45f83f1-kube-api-access-b5ggf\") pod \"maas-api-revoke-sub-c44e4d6c-fflqb\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:40.051496 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:40.051470 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:40.323604 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:40.323557 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb"] Nov 01 23:07:40.326399 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:07:40.326373 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75d385f_258f_42e0_815b_375ba45f83f1.slice/crio-f2e27a0157273c788afdf22a8587b390ac490d2f933dc1055d12606126b2b056 WatchSource:0}: Error finding container f2e27a0157273c788afdf22a8587b390ac490d2f933dc1055d12606126b2b056: Status 404 returned error can't find the container with id f2e27a0157273c788afdf22a8587b390ac490d2f933dc1055d12606126b2b056 Nov 01 23:07:40.593838 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:40.593771 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" event={"ID":"e75d385f-258f-42e0-815b-375ba45f83f1","Type":"ContainerStarted","Data":"5036d308fd0960263fa1d3d0f03f5071fc73f0f4645b4bd043a05aae438582b1"} Nov 01 23:07:40.593838 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:40.593829 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" event={"ID":"e75d385f-258f-42e0-815b-375ba45f83f1","Type":"ContainerStarted","Data":"f2e27a0157273c788afdf22a8587b390ac490d2f933dc1055d12606126b2b056"} Nov 01 23:07:40.614019 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:40.613975 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" podStartSLOduration=1.613962457 podStartE2EDuration="1.613962457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:40.611334709 +0000 UTC m=+2673.259833943" watchObservedRunningTime="2026-11-01 23:07:40.613962457 +0000 UTC m=+2673.262461692" Nov 01 23:07:41.601516 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:41.601482 6287 generic.go:356] "Generic (PLEG): container finished" podID="e75d385f-258f-42e0-815b-375ba45f83f1" containerID="5036d308fd0960263fa1d3d0f03f5071fc73f0f4645b4bd043a05aae438582b1" exitCode=0 Nov 01 23:07:41.601937 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:41.601541 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" event={"ID":"e75d385f-258f-42e0-815b-375ba45f83f1","Type":"ContainerDied","Data":"5036d308fd0960263fa1d3d0f03f5071fc73f0f4645b4bd043a05aae438582b1"} Nov 01 23:07:42.738695 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.738669 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:42.788196 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.788163 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-xks-ca\") pod \"e75d385f-258f-42e0-815b-375ba45f83f1\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " Nov 01 23:07:42.788362 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.788219 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75d385f-258f-42e0-815b-375ba45f83f1-kube-api-access-b5ggf\" (UniqueName: \"kubernetes.io/projected/e75d385f-258f-42e0-815b-375ba45f83f1-kube-api-access-b5ggf\") pod \"e75d385f-258f-42e0-815b-375ba45f83f1\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " Nov 01 23:07:42.788362 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.788323 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-openshift-ca\") pod \"e75d385f-258f-42e0-815b-375ba45f83f1\" (UID: \"e75d385f-258f-42e0-815b-375ba45f83f1\") " Nov 01 23:07:42.788759 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.788731 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-openshift-ca" pod "e75d385f-258f-42e0-815b-375ba45f83f1" (UID: "e75d385f-258f-42e0-815b-375ba45f83f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:42.790327 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.790303 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-xks-ca" pod "e75d385f-258f-42e0-815b-375ba45f83f1" (UID: "e75d385f-258f-42e0-815b-375ba45f83f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:42.790434 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.790396 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75d385f-258f-42e0-815b-375ba45f83f1-kube-api-access-b5ggf" pod "e75d385f-258f-42e0-815b-375ba45f83f1" (UID: "e75d385f-258f-42e0-815b-375ba45f83f1"). InnerVolumeSpecName "kube-api-access-b5ggf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:42.890065 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.889985 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-xks-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:42.890065 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.890025 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5ggf\" (UniqueName: \"kubernetes.io/projected/e75d385f-258f-42e0-815b-375ba45f83f1-kube-api-access-b5ggf\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:42.890065 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:42.890040 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75d385f-258f-42e0-815b-375ba45f83f1-maas-api-openshift-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:43.613422 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:43.613394 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" Nov 01 23:07:43.613587 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:43.613393 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb" event={"ID":"e75d385f-258f-42e0-815b-375ba45f83f1","Type":"ContainerDied","Data":"f2e27a0157273c788afdf22a8587b390ac490d2f933dc1055d12606126b2b056"} Nov 01 23:07:43.613587 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:43.613523 6287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2e27a0157273c788afdf22a8587b390ac490d2f933dc1055d12606126b2b056" Nov 01 23:07:45.587164 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:07:45.587089 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:07:46.021092 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:46.021057 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6"] Nov 01 23:07:46.024235 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:46.024203 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ww8k6"] Nov 01 23:07:47.469363 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:47.469328 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5b24c1-9760-4d1e-9766-b6ddc1a369a5" path="/var/lib/kubelet/pods/9d5b24c1-9760-4d1e-9766-b6ddc1a369a5/volumes" Nov 01 23:07:51.706466 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.706419 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr"] Nov 01 23:07:51.707013 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.706991 6287 cpu_manager.go:414] "containerMap: removing container" podUID="e75d385f-258f-42e0-815b-375ba45f83f1" containerName="revoke-keys" Nov 01 23:07:51.707101 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.707015 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75d385f-258f-42e0-815b-375ba45f83f1" containerName="revoke-keys" Nov 01 23:07:51.707205 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.707171 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75d385f-258f-42e0-815b-375ba45f83f1" containerName="revoke-keys" Nov 01 23:07:51.711035 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.711015 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.723076 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.723047 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:07:51.723221 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.723047 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:07:51.725378 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.725352 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr"] Nov 01 23:07:51.771785 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.771752 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-metrics-certs\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.771906 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.771793 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-openshift-service-ca\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.771967 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.771907 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x84xw\" (UniqueName: \"kubernetes.io/projected/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-kube-api-access-x84xw\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.771967 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.771942 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-maas-api-tls\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.784129 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.784086 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 23:07:51.784238 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.784149 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q7x7m\"" Nov 01 23:07:51.784238 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.784160 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 23:07:51.873300 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.873273 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84xw\" (UniqueName: \"kubernetes.io/projected/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-kube-api-access-x84xw\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.873468 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.873310 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-maas-api-tls\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.873468 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.873364 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-metrics-certs\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.873468 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.873387 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-openshift-service-ca\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.874046 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.874021 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-openshift-service-ca\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.875816 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.875791 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-maas-api-tls\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.875935 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.875855 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-metrics-certs\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:51.880974 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:51.880947 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84xw\" (UniqueName: \"kubernetes.io/projected/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-kube-api-access-x84xw\") pod \"maas-api-54c8494dc4-gn4wr\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:52.024950 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:52.024918 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:52.298710 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:52.298678 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr"] Nov 01 23:07:52.300516 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:07:52.300480 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbcfbdd_f90e_46b5_b2ac_0b1dfd15ebc7.slice/crio-b99bb0a435fd52dc5dfc5faad3ed2a4335547eac9eff7c4f63c430678d13ed05 WatchSource:0}: Error finding container b99bb0a435fd52dc5dfc5faad3ed2a4335547eac9eff7c4f63c430678d13ed05: Status 404 returned error can't find the container with id b99bb0a435fd52dc5dfc5faad3ed2a4335547eac9eff7c4f63c430678d13ed05 Nov 01 23:07:52.304998 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:52.304976 6287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:07:52.662175 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:52.662023 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" event={"ID":"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7","Type":"ContainerStarted","Data":"b99bb0a435fd52dc5dfc5faad3ed2a4335547eac9eff7c4f63c430678d13ed05"} Nov 01 23:07:53.668874 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:53.668837 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" event={"ID":"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7","Type":"ContainerStarted","Data":"6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1"} Nov 01 23:07:53.669367 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:53.668902 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:53.688469 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:53.688420 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" podStartSLOduration=2.298270965 podStartE2EDuration="2.688407057s" totalImagesPullingTime="390.136092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:53.687529624 +0000 UTC m=+2686.336028859" watchObservedRunningTime="2026-11-01 23:07:53.688407057 +0000 UTC m=+2686.336906292" Nov 01 23:07:58.707247 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.707210 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t"] Nov 01 23:07:58.714362 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.714333 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.724569 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.724538 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-75893881-openshift-default-dockercfg-ttnk7\"" Nov 01 23:07:58.727177 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.727151 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t"] Nov 01 23:07:58.837378 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.837345 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-credential-socket\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.837550 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.837399 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4pbc\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-kube-api-access-f4pbc\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.837550 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.837509 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-podinfo\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.837653 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.837555 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-data\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.837653 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.837576 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-socket\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.837730 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.837689 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-certs\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.837730 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.837717 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-envoy\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.837806 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.837741 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istiod-ca-cert\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.837806 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.837772 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-token\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939009 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.938969 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-certs\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939009 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939013 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-envoy\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939336 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939040 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istiod-ca-cert\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939336 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939070 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-token\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939336 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939155 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-credential-socket\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939336 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939189 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4pbc\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-kube-api-access-f4pbc\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939336 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939258 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-podinfo\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939336 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939308 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-data\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939341 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-socket\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939510 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-certs\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939630 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-credential-socket\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939675 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939666 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-socket\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939904 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939734 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-data\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.939904 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.939866 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istiod-ca-cert\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.941476 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.941455 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-envoy\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.941820 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.941801 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-podinfo\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.947184 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.947160 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-token\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:58.947265 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:58.947210 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4pbc\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-kube-api-access-f4pbc\") pod \"e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:59.028254 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.028230 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:07:59.274296 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.274258 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr"] Nov 01 23:07:59.274550 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.274507 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" podUID="dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" containerName="maas-api" containerID="cri-o://6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1" gracePeriod=30 Nov 01 23:07:59.280303 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.280239 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:59.309690 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.309662 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t"] Nov 01 23:07:59.313080 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.313056 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65"] Nov 01 23:07:59.344487 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:07:59.344460 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f2cd9d_9e35_476f_af7e_a8d874a5fd6c.slice/crio-932454fd0ccb8ac3e0176c607007e2bcbaf636e0c19c16eeb9590b8f462f4e2a WatchSource:0}: Error finding container 932454fd0ccb8ac3e0176c607007e2bcbaf636e0c19c16eeb9590b8f462f4e2a: Status 404 returned error can't find the container with id 932454fd0ccb8ac3e0176c607007e2bcbaf636e0c19c16eeb9590b8f462f4e2a Nov 01 23:07:59.344867 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.344847 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65"] Nov 01 23:07:59.344979 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.344964 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.380843 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.380807 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:07:59.380970 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.380885 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:07:59.381043 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.380968 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:07:59.445923 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.445896 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-openshift-service-ca\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.446030 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.445941 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-maas-api-tls\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.446074 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.446026 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-metrics-certs\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.446127 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.446085 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trjzz\" (UniqueName: \"kubernetes.io/projected/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-kube-api-access-trjzz\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.547534 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.547324 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-maas-api-tls\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.547534 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.547378 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-metrics-certs\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.547534 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.547437 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjzz\" (UniqueName: \"kubernetes.io/projected/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-kube-api-access-trjzz\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.547534 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.547513 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-openshift-service-ca\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.548293 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.548267 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-openshift-service-ca\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.551475 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.551430 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-metrics-certs\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.551587 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.551470 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-maas-api-tls\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.559781 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.559733 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjzz\" (UniqueName: \"kubernetes.io/projected/4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949-kube-api-access-trjzz\") pod \"maas-api-5fdf88964b-zvq65\" (UID: \"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949\") " pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.605004 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.604977 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:59.697854 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.697814 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:07:59.707233 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.707197 6287 generic.go:356] "Generic (PLEG): container finished" podID="dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" containerID="6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1" exitCode=0 Nov 01 23:07:59.707383 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.707260 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" Nov 01 23:07:59.707383 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.707359 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" event={"ID":"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7","Type":"ContainerDied","Data":"6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1"} Nov 01 23:07:59.707819 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.707407 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr" event={"ID":"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7","Type":"ContainerDied","Data":"b99bb0a435fd52dc5dfc5faad3ed2a4335547eac9eff7c4f63c430678d13ed05"} Nov 01 23:07:59.707819 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.707440 6287 scope.go:122] "RemoveContainer" containerID="6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1" Nov 01 23:07:59.709513 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.709489 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" event={"ID":"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c","Type":"ContainerStarted","Data":"a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276"} Nov 01 23:07:59.709616 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.709535 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" event={"ID":"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c","Type":"ContainerStarted","Data":"932454fd0ccb8ac3e0176c607007e2bcbaf636e0c19c16eeb9590b8f462f4e2a"} Nov 01 23:07:59.718838 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.718817 6287 scope.go:122] "RemoveContainer" containerID="6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1" Nov 01 23:07:59.719161 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:07:59.719140 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1\": container with ID starting with 6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1 not found: ID does not exist" containerID="6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1" Nov 01 23:07:59.719286 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.719170 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1"} err="failed to get container status \"6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1\": rpc error: code = NotFound desc = could not find container \"6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1\": container with ID starting with 6c5e30347cc93d004974f402a4f8f0ef45a336e17bc227dc4d98a0c7a3cf9fa1 not found: ID does not exist" Nov 01 23:07:59.733462 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.733406 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" podStartSLOduration=1.733391632 podStartE2EDuration="1.733391632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:59.731272136 +0000 UTC m=+2692.379771382" watchObservedRunningTime="2026-11-01 23:07:59.733391632 +0000 UTC m=+2692.381890866" Nov 01 23:07:59.749716 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.749681 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-metrics-certs\") pod \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " Nov 01 23:07:59.749877 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.749823 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-kube-api-access-x84xw\" (UniqueName: \"kubernetes.io/projected/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-kube-api-access-x84xw\") pod \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " Nov 01 23:07:59.749931 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.749870 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-openshift-service-ca\") pod \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " Nov 01 23:07:59.749931 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.749905 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-maas-api-tls\") pod \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\" (UID: \"dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7\") " Nov 01 23:07:59.751382 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.751353 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-openshift-service-ca" pod "dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" (UID: "dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:59.754705 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.754674 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-maas-api-tls" pod "dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" (UID: "dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:59.755553 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.755504 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-metrics-certs" pod "dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" (UID: "dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:59.756661 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.756633 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-kube-api-access-x84xw" pod "dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" (UID: "dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7"). InnerVolumeSpecName "kube-api-access-x84xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:59.851269 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.851136 6287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-metrics-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:59.851269 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.851177 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x84xw\" (UniqueName: \"kubernetes.io/projected/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-kube-api-access-x84xw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:59.851269 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.851192 6287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-openshift-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:59.851269 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.851209 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7-maas-api-tls\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:59.979488 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:07:59.979455 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65"] Nov 01 23:07:59.981166 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:07:59.981136 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6e458f_f5b8_4e9f_8cfb_2f7dceca8949.slice/crio-5bfc118e6beef75f12519bb19c4b5e23b4827c9a4bd225ce7690dcfb624bd401 WatchSource:0}: Error finding container 5bfc118e6beef75f12519bb19c4b5e23b4827c9a4bd225ce7690dcfb624bd401: Status 404 returned error can't find the container with id 5bfc118e6beef75f12519bb19c4b5e23b4827c9a4bd225ce7690dcfb624bd401 Nov 01 23:08:00.029025 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.028999 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:00.036670 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.034443 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:00.036787 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.036664 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr"] Nov 01 23:08:00.040380 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.040357 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-54c8494dc4-gn4wr"] Nov 01 23:08:00.716787 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.716696 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" event={"ID":"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949","Type":"ContainerStarted","Data":"9701781983dd8dab7932c675cf6a92d5992eb8acd776d6bf13455a388aff3c64"} Nov 01 23:08:00.716787 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.716754 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" event={"ID":"4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949","Type":"ContainerStarted","Data":"5bfc118e6beef75f12519bb19c4b5e23b4827c9a4bd225ce7690dcfb624bd401"} Nov 01 23:08:00.717320 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.716889 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:08:00.718047 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.718032 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:00.719146 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.719127 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:00.735859 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:00.735814 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" podStartSLOduration=1.327011204 podStartE2EDuration="1.735802667s" totalImagesPullingTime="408.791463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:00.73422422 +0000 UTC m=+2693.382723455" watchObservedRunningTime="2026-11-01 23:08:00.735802667 +0000 UTC m=+2693.384301901" Nov 01 23:08:01.470090 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:01.470053 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" path="/var/lib/kubelet/pods/dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7/volumes" Nov 01 23:08:02.553808 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.553775 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs"] Nov 01 23:08:02.554236 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.554201 6287 cpu_manager.go:414] "containerMap: removing container" podUID="dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" containerName="maas-api" Nov 01 23:08:02.554236 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.554213 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" containerName="maas-api" Nov 01 23:08:02.554330 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.554284 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbcfbdd-f90e-46b5-b2ac-0b1dfd15ebc7" containerName="maas-api" Nov 01 23:08:02.556364 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.556348 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:02.563343 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.563221 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-75893881-dockercfg-g6dxf\"" Nov 01 23:08:02.577226 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.577198 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-75893881\"" Nov 01 23:08:02.578256 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.577571 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c"] Nov 01 23:08:02.582940 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.582913 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs"] Nov 01 23:08:02.583033 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.583004 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:02.598321 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.598295 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c"] Nov 01 23:08:02.679566 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.679531 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02299896-40f0-40b6-8e52-1973d76521d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs\" (UID: \"02299896-40f0-40b6-8e52-1973d76521d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:02.679762 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.679668 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271b90c2-49ff-496c-9149-c27c3cebc74a-plugins-config-volume\") pod \"payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c\" (UID: \"271b90c2-49ff-496c-9149-c27c3cebc74a\") " pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:02.679762 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.679727 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6dwb\" (UniqueName: \"kubernetes.io/projected/02299896-40f0-40b6-8e52-1973d76521d9-kube-api-access-s6dwb\") pod \"payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs\" (UID: \"02299896-40f0-40b6-8e52-1973d76521d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:02.679762 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.679760 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktpn\" (UniqueName: \"kubernetes.io/projected/271b90c2-49ff-496c-9149-c27c3cebc74a-kube-api-access-7ktpn\") pod \"payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c\" (UID: \"271b90c2-49ff-496c-9149-c27c3cebc74a\") " pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:02.780823 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.780789 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02299896-40f0-40b6-8e52-1973d76521d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs\" (UID: \"02299896-40f0-40b6-8e52-1973d76521d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:02.781020 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.780867 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271b90c2-49ff-496c-9149-c27c3cebc74a-plugins-config-volume\") pod \"payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c\" (UID: \"271b90c2-49ff-496c-9149-c27c3cebc74a\") " pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:02.781020 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.780902 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6dwb\" (UniqueName: \"kubernetes.io/projected/02299896-40f0-40b6-8e52-1973d76521d9-kube-api-access-s6dwb\") pod \"payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs\" (UID: \"02299896-40f0-40b6-8e52-1973d76521d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:02.781020 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.780931 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktpn\" (UniqueName: \"kubernetes.io/projected/271b90c2-49ff-496c-9149-c27c3cebc74a-kube-api-access-7ktpn\") pod \"payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c\" (UID: \"271b90c2-49ff-496c-9149-c27c3cebc74a\") " pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:02.781863 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.781837 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271b90c2-49ff-496c-9149-c27c3cebc74a-plugins-config-volume\") pod \"payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c\" (UID: \"271b90c2-49ff-496c-9149-c27c3cebc74a\") " pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:02.781921 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.781839 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02299896-40f0-40b6-8e52-1973d76521d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs\" (UID: \"02299896-40f0-40b6-8e52-1973d76521d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:02.790642 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.790616 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktpn\" (UniqueName: \"kubernetes.io/projected/271b90c2-49ff-496c-9149-c27c3cebc74a-kube-api-access-7ktpn\") pod \"payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c\" (UID: \"271b90c2-49ff-496c-9149-c27c3cebc74a\") " pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:02.791583 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.791563 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6dwb\" (UniqueName: \"kubernetes.io/projected/02299896-40f0-40b6-8e52-1973d76521d9-kube-api-access-s6dwb\") pod \"payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs\" (UID: \"02299896-40f0-40b6-8e52-1973d76521d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:02.868567 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.868478 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:02.899294 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:02.899260 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:03.173777 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.173752 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs"] Nov 01 23:08:03.176089 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:08:03.176055 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02299896_40f0_40b6_8e52_1973d76521d9.slice/crio-35df5c9bb07024b3e8384348deec958657fef3c61cc26cdf64d5ad3c1a0bf89f WatchSource:0}: Error finding container 35df5c9bb07024b3e8384348deec958657fef3c61cc26cdf64d5ad3c1a0bf89f: Status 404 returned error can't find the container with id 35df5c9bb07024b3e8384348deec958657fef3c61cc26cdf64d5ad3c1a0bf89f Nov 01 23:08:03.201978 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.201953 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c"] Nov 01 23:08:03.210371 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:08:03.210347 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271b90c2_49ff_496c_9149_c27c3cebc74a.slice/crio-e9e4683ca1a85ad179fa11b59a63ddf3145c1dae0a03ca91193613e77f341b26 WatchSource:0}: Error finding container e9e4683ca1a85ad179fa11b59a63ddf3145c1dae0a03ca91193613e77f341b26: Status 404 returned error can't find the container with id e9e4683ca1a85ad179fa11b59a63ddf3145c1dae0a03ca91193613e77f341b26 Nov 01 23:08:03.742050 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.742001 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" event={"ID":"271b90c2-49ff-496c-9149-c27c3cebc74a","Type":"ContainerStarted","Data":"05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab"} Nov 01 23:08:03.742627 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.742197 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" event={"ID":"271b90c2-49ff-496c-9149-c27c3cebc74a","Type":"ContainerStarted","Data":"e9e4683ca1a85ad179fa11b59a63ddf3145c1dae0a03ca91193613e77f341b26"} Nov 01 23:08:03.742627 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.742285 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:03.744933 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.744908 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" event={"ID":"02299896-40f0-40b6-8e52-1973d76521d9","Type":"ContainerStarted","Data":"f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8"} Nov 01 23:08:03.745088 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.744951 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" event={"ID":"02299896-40f0-40b6-8e52-1973d76521d9","Type":"ContainerStarted","Data":"35df5c9bb07024b3e8384348deec958657fef3c61cc26cdf64d5ad3c1a0bf89f"} Nov 01 23:08:03.745088 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.745041 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:03.763130 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.763066 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" podStartSLOduration=1.763048046 podStartE2EDuration="1.763048046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:03.761767795 +0000 UTC m=+2696.410267031" watchObservedRunningTime="2026-11-01 23:08:03.763048046 +0000 UTC m=+2696.411547393" Nov 01 23:08:03.779201 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:03.779153 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" podStartSLOduration=1.779141079 podStartE2EDuration="1.779141079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:03.777141968 +0000 UTC m=+2696.425641204" watchObservedRunningTime="2026-11-01 23:08:03.779141079 +0000 UTC m=+2696.427640314" Nov 01 23:08:06.731150 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:06.731098 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fdf88964b-zvq65" Nov 01 23:08:14.757227 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:14.757191 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:14.757837 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:14.757265 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:15.593641 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:08:15.593603 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:08:15.719394 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:15.719357 6287 patch_prober.go:29] interesting pod/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 01 23:08:15.719600 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:15.719414 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 01 23:08:15.842086 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:15.842054 6287 generic.go:356] "Generic (PLEG): container finished" podID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerID="a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276" exitCode=0 Nov 01 23:08:15.842524 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:15.842108 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" event={"ID":"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c","Type":"ContainerDied","Data":"a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276"} Nov 01 23:08:15.842524 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:15.842460 6287 scope.go:122] "RemoveContainer" containerID="a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276" Nov 01 23:08:15.843621 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:15.843588 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:08:15.843754 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:15.843659 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:08:15.843754 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:15.843690 6287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"} Nov 01 23:08:16.849282 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:16.849202 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" event={"ID":"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c","Type":"ContainerStarted","Data":"c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7"} Nov 01 23:08:17.028719 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:17.028682 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:17.028947 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:17.028927 6287 patch_prober.go:29] interesting pod/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 01 23:08:17.029007 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:17.028972 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 01 23:08:17.425089 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:17.425061 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 23:08:17.425761 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:17.425744 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 23:08:18.028685 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:18.028641 6287 patch_prober.go:29] interesting pod/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 01 23:08:18.029099 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:18.028710 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 01 23:08:19.029471 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:19.029434 6287 patch_prober.go:29] interesting pod/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 01 23:08:19.029889 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:19.029492 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 01 23:08:19.829404 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:19.829375 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 23:08:19.830083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:19.830064 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 23:08:20.029045 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:20.029006 6287 patch_prober.go:29] interesting pod/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 01 23:08:20.029273 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:20.029084 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 01 23:08:21.028747 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:21.028714 6287 patch_prober.go:29] interesting pod/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" start-of-body= Nov 01 23:08:21.029156 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:21.028780 6287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.19:15021/healthz/ready\": dial tcp 10.133.1.19:15021: connect: connection refused" Nov 01 23:08:22.032910 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:22.032877 6287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:22.033356 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:22.033227 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:22.034091 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:22.034073 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:27.175881 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:27.175847 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c"] Nov 01 23:08:27.194612 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:27.194580 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs"] Nov 01 23:08:32.176273 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.176215 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" podUID="271b90c2-49ff-496c-9149-c27c3cebc74a" containerName="payload-processing" containerID="cri-o://05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab" gracePeriod=25 Nov 01 23:08:32.195773 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.195743 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" podUID="02299896-40f0-40b6-8e52-1973d76521d9" containerName="payload-pre-processing" containerID="cri-o://f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8" gracePeriod=25 Nov 01 23:08:32.440005 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.439949 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:32.484580 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.484558 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:32.572377 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.572338 6287 scope.go:122] "RemoveContainer" containerID="c026d32ff59a4a0ced0b61177fbc94444fa370aa6daea5ddc31cda3333729115" Nov 01 23:08:32.581251 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.581205 6287 scope.go:122] "RemoveContainer" containerID="3103462df0acf863ab3629383d0b2052b3d22f4641611d3502f2b34ce2cf4723" Nov 01 23:08:32.615358 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.615335 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271b90c2-49ff-496c-9149-c27c3cebc74a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271b90c2-49ff-496c-9149-c27c3cebc74a-plugins-config-volume\") pod \"271b90c2-49ff-496c-9149-c27c3cebc74a\" (UID: \"271b90c2-49ff-496c-9149-c27c3cebc74a\") " Nov 01 23:08:32.615470 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.615379 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02299896-40f0-40b6-8e52-1973d76521d9-kube-api-access-s6dwb\" (UniqueName: \"kubernetes.io/projected/02299896-40f0-40b6-8e52-1973d76521d9-kube-api-access-s6dwb\") pod \"02299896-40f0-40b6-8e52-1973d76521d9\" (UID: \"02299896-40f0-40b6-8e52-1973d76521d9\") " Nov 01 23:08:32.615470 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.615424 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02299896-40f0-40b6-8e52-1973d76521d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02299896-40f0-40b6-8e52-1973d76521d9-plugins-config-volume\") pod \"02299896-40f0-40b6-8e52-1973d76521d9\" (UID: \"02299896-40f0-40b6-8e52-1973d76521d9\") " Nov 01 23:08:32.615600 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.615494 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271b90c2-49ff-496c-9149-c27c3cebc74a-kube-api-access-7ktpn\" (UniqueName: \"kubernetes.io/projected/271b90c2-49ff-496c-9149-c27c3cebc74a-kube-api-access-7ktpn\") pod \"271b90c2-49ff-496c-9149-c27c3cebc74a\" (UID: \"271b90c2-49ff-496c-9149-c27c3cebc74a\") " Nov 01 23:08:32.615666 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.615645 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b90c2-49ff-496c-9149-c27c3cebc74a-plugins-config-volume" pod "271b90c2-49ff-496c-9149-c27c3cebc74a" (UID: "271b90c2-49ff-496c-9149-c27c3cebc74a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:32.615834 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.615808 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02299896-40f0-40b6-8e52-1973d76521d9-plugins-config-volume" pod "02299896-40f0-40b6-8e52-1973d76521d9" (UID: "02299896-40f0-40b6-8e52-1973d76521d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:32.615834 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.615831 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271b90c2-49ff-496c-9149-c27c3cebc74a-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:32.617569 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.617547 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271b90c2-49ff-496c-9149-c27c3cebc74a-kube-api-access-7ktpn" pod "271b90c2-49ff-496c-9149-c27c3cebc74a" (UID: "271b90c2-49ff-496c-9149-c27c3cebc74a"). InnerVolumeSpecName "kube-api-access-7ktpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:32.617751 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.617732 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02299896-40f0-40b6-8e52-1973d76521d9-kube-api-access-s6dwb" pod "02299896-40f0-40b6-8e52-1973d76521d9" (UID: "02299896-40f0-40b6-8e52-1973d76521d9"). InnerVolumeSpecName "kube-api-access-s6dwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:32.716406 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.716340 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktpn\" (UniqueName: \"kubernetes.io/projected/271b90c2-49ff-496c-9149-c27c3cebc74a-kube-api-access-7ktpn\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:32.716406 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.716364 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6dwb\" (UniqueName: \"kubernetes.io/projected/02299896-40f0-40b6-8e52-1973d76521d9-kube-api-access-s6dwb\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:32.716406 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.716374 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02299896-40f0-40b6-8e52-1973d76521d9-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:32.939409 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.939375 6287 generic.go:356] "Generic (PLEG): container finished" podID="02299896-40f0-40b6-8e52-1973d76521d9" containerID="f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8" exitCode=0 Nov 01 23:08:32.939593 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.939419 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" Nov 01 23:08:32.939593 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.939446 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" event={"ID":"02299896-40f0-40b6-8e52-1973d76521d9","Type":"ContainerDied","Data":"f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8"} Nov 01 23:08:32.939593 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.939500 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs" event={"ID":"02299896-40f0-40b6-8e52-1973d76521d9","Type":"ContainerDied","Data":"35df5c9bb07024b3e8384348deec958657fef3c61cc26cdf64d5ad3c1a0bf89f"} Nov 01 23:08:32.939593 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.939533 6287 scope.go:122] "RemoveContainer" containerID="f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8" Nov 01 23:08:32.941009 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.940977 6287 generic.go:356] "Generic (PLEG): container finished" podID="271b90c2-49ff-496c-9149-c27c3cebc74a" containerID="05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab" exitCode=0 Nov 01 23:08:32.941163 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.941019 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" Nov 01 23:08:32.941163 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.941063 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" event={"ID":"271b90c2-49ff-496c-9149-c27c3cebc74a","Type":"ContainerDied","Data":"05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab"} Nov 01 23:08:32.941163 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.941133 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c" event={"ID":"271b90c2-49ff-496c-9149-c27c3cebc74a","Type":"ContainerDied","Data":"e9e4683ca1a85ad179fa11b59a63ddf3145c1dae0a03ca91193613e77f341b26"} Nov 01 23:08:32.950241 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.950223 6287 scope.go:122] "RemoveContainer" containerID="f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8" Nov 01 23:08:32.950583 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:08:32.950489 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8\": container with ID starting with f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8 not found: ID does not exist" containerID="f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8" Nov 01 23:08:32.950583 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.950520 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8"} err="failed to get container status \"f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8\": rpc error: code = NotFound desc = could not find container \"f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8\": container with ID starting with f099761233636a445f91fece9e2eea7f61ddb3290fcd48183d53cd994daa72d8 not found: ID does not exist" Nov 01 23:08:32.950583 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.950538 6287 scope.go:122] "RemoveContainer" containerID="05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab" Nov 01 23:08:32.958641 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.958618 6287 scope.go:122] "RemoveContainer" containerID="05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab" Nov 01 23:08:32.958896 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:08:32.958878 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab\": container with ID starting with 05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab not found: ID does not exist" containerID="05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab" Nov 01 23:08:32.958975 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.958908 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab"} err="failed to get container status \"05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab\": rpc error: code = NotFound desc = could not find container \"05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab\": container with ID starting with 05d9e1fc1b872d59c22b04af3df3f27009b6b0b67fe6673156c106b4c0cdb9ab not found: ID does not exist" Nov 01 23:08:32.962621 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.962600 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs"] Nov 01 23:08:32.965874 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.965852 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75893881-7b6dcd8fd7-cmtcs"] Nov 01 23:08:32.975259 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.975212 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c"] Nov 01 23:08:32.979160 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:32.979141 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75893881-67b9bdd49c-zwk7c"] Nov 01 23:08:33.111741 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:33.111699 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t"] Nov 01 23:08:33.112066 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:33.112025 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" containerID="cri-o://c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7" gracePeriod=30 Nov 01 23:08:33.469078 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:33.469049 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02299896-40f0-40b6-8e52-1973d76521d9" path="/var/lib/kubelet/pods/02299896-40f0-40b6-8e52-1973d76521d9/volumes" Nov 01 23:08:33.469478 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:33.469439 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271b90c2-49ff-496c-9149-c27c3cebc74a" path="/var/lib/kubelet/pods/271b90c2-49ff-496c-9149-c27c3cebc74a/volumes" Nov 01 23:08:37.035433 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:37.035388 6287 patch_prober.go:29] interesting pod/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:08:37.035816 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:37.035461 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:08:38.367099 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.367075 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:38.467431 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467349 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-certs\") pod \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " Nov 01 23:08:38.467431 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467398 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-data\") pod \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " Nov 01 23:08:38.467431 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467432 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-podinfo\") pod \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " Nov 01 23:08:38.467732 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467458 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-token\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-token\") pod \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " Nov 01 23:08:38.467732 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467481 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istiod-ca-cert\") pod \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " Nov 01 23:08:38.467732 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467556 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-kube-api-access-f4pbc\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-kube-api-access-f4pbc\") pod \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " Nov 01 23:08:38.467732 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467607 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-envoy\") pod \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " Nov 01 23:08:38.467732 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467640 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-credential-socket\") pod \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " Nov 01 23:08:38.467732 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467640 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-certs" pod "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" (UID: "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:38.467732 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467666 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-socket\") pod \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\" (UID: \"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c\") " Nov 01 23:08:38.468165 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467810 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-data" pod "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" (UID: "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:38.468165 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467905 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-credential-socket" pod "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" (UID: "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:38.468165 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.467990 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-socket" pod "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" (UID: "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:38.468165 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.468066 6287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-credential-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:38.468165 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.468082 6287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-socket\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:38.468165 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.468094 6287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-workload-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:38.468165 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.468104 6287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-data\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:38.468581 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.468302 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istiod-ca-cert" pod "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" (UID: "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:38.469871 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.469840 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-podinfo" pod "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" (UID: "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:08:38.469995 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.469941 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-token" pod "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" (UID: "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:38.469995 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.469940 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-kube-api-access-f4pbc" pod "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" (UID: "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c"). InnerVolumeSpecName "kube-api-access-f4pbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:38.469995 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.469956 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-envoy" pod "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" (UID: "26f2cd9d-9e35-476f-af7e-a8d874a5fd6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:38.568930 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.568900 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4pbc\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-kube-api-access-f4pbc\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:38.568930 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.568928 6287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-envoy\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:38.569103 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.568938 6287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-podinfo\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:38.569103 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.568947 6287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istio-token\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:38.569103 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.568955 6287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c-istiod-ca-cert\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:38.826600 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.826559 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86"] Nov 01 23:08:38.827015 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.826996 6287 cpu_manager.go:414] "containerMap: removing container" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" Nov 01 23:08:38.827138 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.827017 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" Nov 01 23:08:38.827138 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.827036 6287 cpu_manager.go:414] "containerMap: removing container" podUID="02299896-40f0-40b6-8e52-1973d76521d9" containerName="payload-pre-processing" Nov 01 23:08:38.827138 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.827043 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02299896-40f0-40b6-8e52-1973d76521d9" containerName="payload-pre-processing" Nov 01 23:08:38.827138 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.827072 6287 cpu_manager.go:414] "containerMap: removing container" podUID="271b90c2-49ff-496c-9149-c27c3cebc74a" containerName="payload-processing" Nov 01 23:08:38.827138 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.827080 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271b90c2-49ff-496c-9149-c27c3cebc74a" containerName="payload-processing" Nov 01 23:08:38.827412 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.827239 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" Nov 01 23:08:38.827412 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.827254 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="02299896-40f0-40b6-8e52-1973d76521d9" containerName="payload-pre-processing" Nov 01 23:08:38.827412 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.827269 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="271b90c2-49ff-496c-9149-c27c3cebc74a" containerName="payload-processing" Nov 01 23:08:38.827412 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.827282 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" Nov 01 23:08:38.830633 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.830612 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.849218 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.843996 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-4b45d6-serving-cert\"" Nov 01 23:08:38.849218 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.847455 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86"] Nov 01 23:08:38.871367 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.871339 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555280ff-5432-484c-a622-640625058dce-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.871524 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.871373 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-metrics-certs\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.871591 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.871555 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.871651 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.871638 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjd44\" (UniqueName: \"kubernetes.io/projected/555280ff-5432-484c-a622-640625058dce-kube-api-access-kjd44\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.902734 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.902706 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4"] Nov 01 23:08:38.903102 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.903088 6287 cpu_manager.go:414] "containerMap: removing container" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" Nov 01 23:08:38.903174 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.903103 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerName="istio-proxy" Nov 01 23:08:38.906282 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.906263 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:08:38.916299 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.916272 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-4b45d6\"" Nov 01 23:08:38.918198 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.918174 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4"] Nov 01 23:08:38.939405 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.939379 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-4b45d6-dockercfg-g2qwv\"" Nov 01 23:08:38.947018 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.946996 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc"] Nov 01 23:08:38.950476 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.950460 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:38.958698 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.958675 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc"] Nov 01 23:08:38.972771 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.972738 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnb4\" (UniqueName: \"kubernetes.io/projected/7f49282b-90a8-4ab9-954a-68996c610eea-kube-api-access-nxnb4\") pod \"payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4\" (UID: \"7f49282b-90a8-4ab9-954a-68996c610eea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:08:38.972895 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.972784 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.972895 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.972876 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjd44\" (UniqueName: \"kubernetes.io/projected/555280ff-5432-484c-a622-640625058dce-kube-api-access-kjd44\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.973020 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:08:38.972909 6287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-serving-cert: secret "maas-api-e2e-authcfg-http-4b45d6-serving-cert" not found Nov 01 23:08:38.973020 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.972913 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j42lw\" (UniqueName: \"kubernetes.io/projected/08c2d506-3b3c-40ab-9365-8dc11f484de7-kube-api-access-j42lw\") pod \"payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc\" (UID: \"08c2d506-3b3c-40ab-9365-8dc11f484de7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:38.973020 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.972961 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555280ff-5432-484c-a622-640625058dce-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.973020 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:08:38.972987 6287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls podName:555280ff-5432-484c-a622-640625058dce nodeName:}" failed. No retries permitted until 2026-11-01 23:08:39.472959725 +0000 UTC m=+2732.121458955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls") pod "maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" (UID: "555280ff-5432-484c-a622-640625058dce") : secret "maas-api-e2e-authcfg-http-4b45d6-serving-cert" not found Nov 01 23:08:38.973253 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.973017 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-metrics-certs\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.973253 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.973072 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f49282b-90a8-4ab9-954a-68996c610eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4\" (UID: \"7f49282b-90a8-4ab9-954a-68996c610eea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:08:38.973253 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.973123 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c2d506-3b3c-40ab-9365-8dc11f484de7-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc\" (UID: \"08c2d506-3b3c-40ab-9365-8dc11f484de7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:38.973556 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.973535 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555280ff-5432-484c-a622-640625058dce-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.975897 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.975869 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-metrics-certs\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.981565 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.981538 6287 generic.go:356] "Generic (PLEG): container finished" podID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" containerID="c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7" exitCode=0 Nov 01 23:08:38.981681 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.981585 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" event={"ID":"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c","Type":"ContainerDied","Data":"c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7"} Nov 01 23:08:38.981681 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.981608 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" Nov 01 23:08:38.981681 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.981632 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t" event={"ID":"26f2cd9d-9e35-476f-af7e-a8d874a5fd6c","Type":"ContainerDied","Data":"932454fd0ccb8ac3e0176c607007e2bcbaf636e0c19c16eeb9590b8f462f4e2a"} Nov 01 23:08:38.981681 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.981663 6287 scope.go:122] "RemoveContainer" containerID="c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7" Nov 01 23:08:38.982949 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.982927 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjd44\" (UniqueName: \"kubernetes.io/projected/555280ff-5432-484c-a622-640625058dce-kube-api-access-kjd44\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:38.992084 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:38.992065 6287 scope.go:122] "RemoveContainer" containerID="a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276" Nov 01 23:08:39.000700 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.000678 6287 scope.go:122] "RemoveContainer" containerID="c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7" Nov 01 23:08:39.001006 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:08:39.000984 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7\": container with ID starting with c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7 not found: ID does not exist" containerID="c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7" Nov 01 23:08:39.001080 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.001014 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7"} err="failed to get container status \"c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7\": rpc error: code = NotFound desc = could not find container \"c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7\": container with ID starting with c9a87c37d27dc16f9bea14ee0b495844f1d791aff9c1f0ca7459607a8c3c80d7 not found: ID does not exist" Nov 01 23:08:39.001080 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.001032 6287 scope.go:122] "RemoveContainer" containerID="a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276" Nov 01 23:08:39.001345 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:08:39.001320 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276\": container with ID starting with a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276 not found: ID does not exist" containerID="a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276" Nov 01 23:08:39.001391 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.001355 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276"} err="failed to get container status \"a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276\": rpc error: code = NotFound desc = could not find container \"a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276\": container with ID starting with a73b09731c018e71941a33819048f8a7c9e4ef208230f3686d4773cc0f00c276 not found: ID does not exist" Nov 01 23:08:39.015228 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.015202 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t"] Nov 01 23:08:39.032395 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.031990 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-75893881-openshift-default-6c99c7ddc9-bl58t"] Nov 01 23:08:39.074093 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.074056 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f49282b-90a8-4ab9-954a-68996c610eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4\" (UID: \"7f49282b-90a8-4ab9-954a-68996c610eea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:08:39.074093 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.074093 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c2d506-3b3c-40ab-9365-8dc11f484de7-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc\" (UID: \"08c2d506-3b3c-40ab-9365-8dc11f484de7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:39.074394 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.074249 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnb4\" (UniqueName: \"kubernetes.io/projected/7f49282b-90a8-4ab9-954a-68996c610eea-kube-api-access-nxnb4\") pod \"payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4\" (UID: \"7f49282b-90a8-4ab9-954a-68996c610eea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:08:39.074394 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.074374 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j42lw\" (UniqueName: \"kubernetes.io/projected/08c2d506-3b3c-40ab-9365-8dc11f484de7-kube-api-access-j42lw\") pod \"payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc\" (UID: \"08c2d506-3b3c-40ab-9365-8dc11f484de7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:39.074901 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.074876 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c2d506-3b3c-40ab-9365-8dc11f484de7-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc\" (UID: \"08c2d506-3b3c-40ab-9365-8dc11f484de7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:39.075026 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.074901 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f49282b-90a8-4ab9-954a-68996c610eea-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4\" (UID: \"7f49282b-90a8-4ab9-954a-68996c610eea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:08:39.082855 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.082790 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnb4\" (UniqueName: \"kubernetes.io/projected/7f49282b-90a8-4ab9-954a-68996c610eea-kube-api-access-nxnb4\") pod \"payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4\" (UID: \"7f49282b-90a8-4ab9-954a-68996c610eea\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:08:39.083003 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.082928 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j42lw\" (UniqueName: \"kubernetes.io/projected/08c2d506-3b3c-40ab-9365-8dc11f484de7-kube-api-access-j42lw\") pod \"payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc\" (UID: \"08c2d506-3b3c-40ab-9365-8dc11f484de7\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:39.221475 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.221442 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:08:39.264981 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.264948 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:39.469588 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.469553 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f2cd9d-9e35-476f-af7e-a8d874a5fd6c" path="/var/lib/kubelet/pods/26f2cd9d-9e35-476f-af7e-a8d874a5fd6c/volumes" Nov 01 23:08:39.478951 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.478913 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:39.481443 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.481422 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:39.498526 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.498503 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4"] Nov 01 23:08:39.500720 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:08:39.500699 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f49282b_90a8_4ab9_954a_68996c610eea.slice/crio-dd446cd044ffd25e01e39f1823cb1fad60e371042954df373daa617dfa27b87f WatchSource:0}: Error finding container dd446cd044ffd25e01e39f1823cb1fad60e371042954df373daa617dfa27b87f: Status 404 returned error can't find the container with id dd446cd044ffd25e01e39f1823cb1fad60e371042954df373daa617dfa27b87f Nov 01 23:08:39.562543 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.562521 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc"] Nov 01 23:08:39.563810 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:08:39.563784 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c2d506_3b3c_40ab_9365_8dc11f484de7.slice/crio-1469d9874ae7cc664660adf901d25d5470a8664d1933255215472af6d6af95b0 WatchSource:0}: Error finding container 1469d9874ae7cc664660adf901d25d5470a8664d1933255215472af6d6af95b0: Status 404 returned error can't find the container with id 1469d9874ae7cc664660adf901d25d5470a8664d1933255215472af6d6af95b0 Nov 01 23:08:39.752591 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.752559 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:39.990275 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.990239 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" event={"ID":"7f49282b-90a8-4ab9-954a-68996c610eea","Type":"ContainerStarted","Data":"98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14"} Nov 01 23:08:39.990481 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.990415 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:08:39.990481 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.990451 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" event={"ID":"7f49282b-90a8-4ab9-954a-68996c610eea","Type":"ContainerStarted","Data":"dd446cd044ffd25e01e39f1823cb1fad60e371042954df373daa617dfa27b87f"} Nov 01 23:08:39.991811 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.991782 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" event={"ID":"08c2d506-3b3c-40ab-9365-8dc11f484de7","Type":"ContainerStarted","Data":"f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd"} Nov 01 23:08:39.991876 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.991839 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:39.991876 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:39.991856 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" event={"ID":"08c2d506-3b3c-40ab-9365-8dc11f484de7","Type":"ContainerStarted","Data":"1469d9874ae7cc664660adf901d25d5470a8664d1933255215472af6d6af95b0"} Nov 01 23:08:40.032809 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:40.020291 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" podStartSLOduration=2.020273712 podStartE2EDuration="2.020273712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:40.007786528 +0000 UTC m=+2732.656285768" watchObservedRunningTime="2026-11-01 23:08:40.020273712 +0000 UTC m=+2732.668772944" Nov 01 23:08:40.033651 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:40.033603 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" podStartSLOduration=2.033589438 podStartE2EDuration="2.033589438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:40.033309751 +0000 UTC m=+2732.681808989" watchObservedRunningTime="2026-11-01 23:08:40.033589438 +0000 UTC m=+2732.682088673" Nov 01 23:08:40.039103 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:40.039075 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86"] Nov 01 23:08:40.042768 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:08:40.042744 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555280ff_5432_484c_a622_640625058dce.slice/crio-020812127acaa61ca4a999d328723c41dea30018a74529195beb7a28bf09ea8f WatchSource:0}: Error finding container 020812127acaa61ca4a999d328723c41dea30018a74529195beb7a28bf09ea8f: Status 404 returned error can't find the container with id 020812127acaa61ca4a999d328723c41dea30018a74529195beb7a28bf09ea8f Nov 01 23:08:40.998394 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:40.998346 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" event={"ID":"555280ff-5432-484c-a622-640625058dce","Type":"ContainerStarted","Data":"19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4"} Nov 01 23:08:40.998812 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:40.998414 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" event={"ID":"555280ff-5432-484c-a622-640625058dce","Type":"ContainerStarted","Data":"020812127acaa61ca4a999d328723c41dea30018a74529195beb7a28bf09ea8f"} Nov 01 23:08:40.998812 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:40.998790 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:41.018804 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:41.018747 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" podStartSLOduration=2.612253332 podStartE2EDuration="3.018729053s" totalImagesPullingTime="406.475721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:41.015917684 +0000 UTC m=+2733.664416920" watchObservedRunningTime="2026-11-01 23:08:41.018729053 +0000 UTC m=+2733.667228289" Nov 01 23:08:45.586749 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:08:45.586712 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:08:48.017771 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:48.017743 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:08:51.002202 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:51.002173 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:08:51.002624 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:08:51.002246 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:09:15.595290 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:09:15.595249 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:09:32.616768 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:09:32.616733 6287 scope.go:122] "RemoveContainer" containerID="890f371b1e503133cb442306aac7569e4d80154c9a8f6d2326d239b6f4c77377" Nov 01 23:09:38.041842 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:09:38.041795 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr"] Nov 01 23:09:38.043997 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:09:38.043968 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wrpfr"] Nov 01 23:09:39.469224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:09:39.469189 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d8422c-12e6-48cf-956e-fa6337896689" path="/var/lib/kubelet/pods/f9d8422c-12e6-48cf-956e-fa6337896689/volumes" Nov 01 23:09:45.586950 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:09:45.586905 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:09:59.023739 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:09:59.023656 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7"] Nov 01 23:09:59.025513 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:09:59.025486 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gf9s7"] Nov 01 23:09:59.470430 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:09:59.470345 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48776640-c170-4b11-998f-bfc52c7b703f" path="/var/lib/kubelet/pods/48776640-c170-4b11-998f-bfc52c7b703f/volumes" Nov 01 23:10:15.587267 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:10:15.587224 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:10:25.301648 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:25.301608 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc"] Nov 01 23:10:25.311021 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:25.310972 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4"] Nov 01 23:10:30.302954 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.302907 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" podUID="08c2d506-3b3c-40ab-9365-8dc11f484de7" containerName="payload-processing" containerID="cri-o://f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd" gracePeriod=25 Nov 01 23:10:30.312131 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.312086 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" podUID="7f49282b-90a8-4ab9-954a-68996c610eea" containerName="payload-pre-processing" containerID="cri-o://98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14" gracePeriod=25 Nov 01 23:10:30.553309 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.553249 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:10:30.584270 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.584246 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:10:30.598899 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.598871 6287 generic.go:356] "Generic (PLEG): container finished" podID="7f49282b-90a8-4ab9-954a-68996c610eea" containerID="98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14" exitCode=0 Nov 01 23:10:30.599033 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.598982 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" Nov 01 23:10:30.599033 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.599013 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" event={"ID":"7f49282b-90a8-4ab9-954a-68996c610eea","Type":"ContainerDied","Data":"98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14"} Nov 01 23:10:30.599192 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.599058 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4" event={"ID":"7f49282b-90a8-4ab9-954a-68996c610eea","Type":"ContainerDied","Data":"dd446cd044ffd25e01e39f1823cb1fad60e371042954df373daa617dfa27b87f"} Nov 01 23:10:30.599192 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.599090 6287 scope.go:122] "RemoveContainer" containerID="98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14" Nov 01 23:10:30.600443 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.600419 6287 generic.go:356] "Generic (PLEG): container finished" podID="08c2d506-3b3c-40ab-9365-8dc11f484de7" containerID="f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd" exitCode=0 Nov 01 23:10:30.600619 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.600454 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" Nov 01 23:10:30.600619 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.600514 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" event={"ID":"08c2d506-3b3c-40ab-9365-8dc11f484de7","Type":"ContainerDied","Data":"f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd"} Nov 01 23:10:30.600619 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.600555 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc" event={"ID":"08c2d506-3b3c-40ab-9365-8dc11f484de7","Type":"ContainerDied","Data":"1469d9874ae7cc664660adf901d25d5470a8664d1933255215472af6d6af95b0"} Nov 01 23:10:30.609952 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.609934 6287 scope.go:122] "RemoveContainer" containerID="98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14" Nov 01 23:10:30.610284 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:10:30.610262 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14\": container with ID starting with 98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14 not found: ID does not exist" containerID="98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14" Nov 01 23:10:30.610381 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.610301 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14"} err="failed to get container status \"98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14\": rpc error: code = NotFound desc = could not find container \"98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14\": container with ID starting with 98b0ff4b7055e1b21dbdc475ce809f711870729ab9e54125bd96de73bc428f14 not found: ID does not exist" Nov 01 23:10:30.610381 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.610324 6287 scope.go:122] "RemoveContainer" containerID="f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd" Nov 01 23:10:30.618737 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.618712 6287 scope.go:122] "RemoveContainer" containerID="f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd" Nov 01 23:10:30.619067 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:10:30.619033 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd\": container with ID starting with f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd not found: ID does not exist" containerID="f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd" Nov 01 23:10:30.619173 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.619079 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd"} err="failed to get container status \"f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd\": rpc error: code = NotFound desc = could not find container \"f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd\": container with ID starting with f10a5b64adbb254ed75bb21ef69dd35135ed2b57abc1aff8eb6ff9ef48c3becd not found: ID does not exist" Nov 01 23:10:30.631417 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.631395 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c2d506-3b3c-40ab-9365-8dc11f484de7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c2d506-3b3c-40ab-9365-8dc11f484de7-plugins-config-volume\") pod \"08c2d506-3b3c-40ab-9365-8dc11f484de7\" (UID: \"08c2d506-3b3c-40ab-9365-8dc11f484de7\") " Nov 01 23:10:30.631533 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.631518 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c2d506-3b3c-40ab-9365-8dc11f484de7-kube-api-access-j42lw\" (UniqueName: \"kubernetes.io/projected/08c2d506-3b3c-40ab-9365-8dc11f484de7-kube-api-access-j42lw\") pod \"08c2d506-3b3c-40ab-9365-8dc11f484de7\" (UID: \"08c2d506-3b3c-40ab-9365-8dc11f484de7\") " Nov 01 23:10:30.631728 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.631710 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c2d506-3b3c-40ab-9365-8dc11f484de7-plugins-config-volume" pod "08c2d506-3b3c-40ab-9365-8dc11f484de7" (UID: "08c2d506-3b3c-40ab-9365-8dc11f484de7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:30.631831 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.631816 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c2d506-3b3c-40ab-9365-8dc11f484de7-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.633584 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.633565 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c2d506-3b3c-40ab-9365-8dc11f484de7-kube-api-access-j42lw" pod "08c2d506-3b3c-40ab-9365-8dc11f484de7" (UID: "08c2d506-3b3c-40ab-9365-8dc11f484de7"). InnerVolumeSpecName "kube-api-access-j42lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:30.732615 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.732578 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f49282b-90a8-4ab9-954a-68996c610eea-kube-api-access-nxnb4\" (UniqueName: \"kubernetes.io/projected/7f49282b-90a8-4ab9-954a-68996c610eea-kube-api-access-nxnb4\") pod \"7f49282b-90a8-4ab9-954a-68996c610eea\" (UID: \"7f49282b-90a8-4ab9-954a-68996c610eea\") " Nov 01 23:10:30.732744 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.732656 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f49282b-90a8-4ab9-954a-68996c610eea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f49282b-90a8-4ab9-954a-68996c610eea-plugins-config-volume\") pod \"7f49282b-90a8-4ab9-954a-68996c610eea\" (UID: \"7f49282b-90a8-4ab9-954a-68996c610eea\") " Nov 01 23:10:30.732976 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.732954 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f49282b-90a8-4ab9-954a-68996c610eea-plugins-config-volume" pod "7f49282b-90a8-4ab9-954a-68996c610eea" (UID: "7f49282b-90a8-4ab9-954a-68996c610eea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:30.733162 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.733142 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j42lw\" (UniqueName: \"kubernetes.io/projected/08c2d506-3b3c-40ab-9365-8dc11f484de7-kube-api-access-j42lw\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.733233 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.733167 6287 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f49282b-90a8-4ab9-954a-68996c610eea-plugins-config-volume\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.734502 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.734483 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f49282b-90a8-4ab9-954a-68996c610eea-kube-api-access-nxnb4" pod "7f49282b-90a8-4ab9-954a-68996c610eea" (UID: "7f49282b-90a8-4ab9-954a-68996c610eea"). InnerVolumeSpecName "kube-api-access-nxnb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:30.834693 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.834633 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxnb4\" (UniqueName: \"kubernetes.io/projected/7f49282b-90a8-4ab9-954a-68996c610eea-kube-api-access-nxnb4\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.924369 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.924340 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc"] Nov 01 23:10:30.928672 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.928651 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-4b45d6-589fb4dcdf-ht9xc"] Nov 01 23:10:30.939286 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.939265 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4"] Nov 01 23:10:30.945368 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:30.945345 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-4b45d6-5b8d58589f-crpm4"] Nov 01 23:10:31.270443 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.270399 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86"] Nov 01 23:10:31.270727 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.270697 6287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" podUID="555280ff-5432-484c-a622-640625058dce" containerName="maas-api" containerID="cri-o://19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4" gracePeriod=30 Nov 01 23:10:31.471326 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.471293 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c2d506-3b3c-40ab-9365-8dc11f484de7" path="/var/lib/kubelet/pods/08c2d506-3b3c-40ab-9365-8dc11f484de7/volumes" Nov 01 23:10:31.471800 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.471781 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f49282b-90a8-4ab9-954a-68996c610eea" path="/var/lib/kubelet/pods/7f49282b-90a8-4ab9-954a-68996c610eea/volumes" Nov 01 23:10:31.524925 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.524872 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:10:31.608872 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.608840 6287 generic.go:356] "Generic (PLEG): container finished" podID="555280ff-5432-484c-a622-640625058dce" containerID="19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4" exitCode=0 Nov 01 23:10:31.609075 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.608890 6287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" Nov 01 23:10:31.609075 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.608925 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" event={"ID":"555280ff-5432-484c-a622-640625058dce","Type":"ContainerDied","Data":"19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4"} Nov 01 23:10:31.609075 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.608980 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86" event={"ID":"555280ff-5432-484c-a622-640625058dce","Type":"ContainerDied","Data":"020812127acaa61ca4a999d328723c41dea30018a74529195beb7a28bf09ea8f"} Nov 01 23:10:31.609075 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.609006 6287 scope.go:122] "RemoveContainer" containerID="19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4" Nov 01 23:10:31.617850 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.617833 6287 scope.go:122] "RemoveContainer" containerID="19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4" Nov 01 23:10:31.618155 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:10:31.618135 6287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4\": container with ID starting with 19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4 not found: ID does not exist" containerID="19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4" Nov 01 23:10:31.618224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.618162 6287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4"} err="failed to get container status \"19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4\": rpc error: code = NotFound desc = could not find container \"19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4\": container with ID starting with 19340202c9b732ebc6f19af903f4b91fde55e6606af7f9fcc9f89f6bec2788e4 not found: ID does not exist" Nov 01 23:10:31.642528 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.642503 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555280ff-5432-484c-a622-640625058dce-kube-api-access-kjd44\" (UniqueName: \"kubernetes.io/projected/555280ff-5432-484c-a622-640625058dce-kube-api-access-kjd44\") pod \"555280ff-5432-484c-a622-640625058dce\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " Nov 01 23:10:31.642630 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.642554 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls\") pod \"555280ff-5432-484c-a622-640625058dce\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " Nov 01 23:10:31.642630 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.642586 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555280ff-5432-484c-a622-640625058dce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555280ff-5432-484c-a622-640625058dce-openshift-service-ca\") pod \"555280ff-5432-484c-a622-640625058dce\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " Nov 01 23:10:31.642630 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.642617 6287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-metrics-certs\") pod \"555280ff-5432-484c-a622-640625058dce\" (UID: \"555280ff-5432-484c-a622-640625058dce\") " Nov 01 23:10:31.642984 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.642962 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555280ff-5432-484c-a622-640625058dce-openshift-service-ca" pod "555280ff-5432-484c-a622-640625058dce" (UID: "555280ff-5432-484c-a622-640625058dce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:31.644713 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.644686 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls" pod "555280ff-5432-484c-a622-640625058dce" (UID: "555280ff-5432-484c-a622-640625058dce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:31.644799 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.644693 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555280ff-5432-484c-a622-640625058dce-kube-api-access-kjd44" pod "555280ff-5432-484c-a622-640625058dce" (UID: "555280ff-5432-484c-a622-640625058dce"). InnerVolumeSpecName "kube-api-access-kjd44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:31.644799 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.644723 6287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-metrics-certs" pod "555280ff-5432-484c-a622-640625058dce" (UID: "555280ff-5432-484c-a622-640625058dce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:31.744293 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.744259 6287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjd44\" (UniqueName: \"kubernetes.io/projected/555280ff-5432-484c-a622-640625058dce-kube-api-access-kjd44\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:31.744293 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.744290 6287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-maas-api-tls\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:31.744469 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.744300 6287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/555280ff-5432-484c-a622-640625058dce-openshift-service-ca\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:31.744469 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.744309 6287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555280ff-5432-484c-a622-640625058dce-metrics-certs\") on node \"ip-10-0-128-55.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:31.931191 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.931161 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86"] Nov 01 23:10:31.935541 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:31.935509 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-4b45d6-79fdf8f479-22d86"] Nov 01 23:10:33.469755 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:33.469720 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555280ff-5432-484c-a622-640625058dce" path="/var/lib/kubelet/pods/555280ff-5432-484c-a622-640625058dce/volumes" Nov 01 23:10:45.587498 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:10:45.587457 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:10:48.611137 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:48.600206 6287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vt5hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:10:48.611137 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:10:48.600307 6287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vt5hm" podUID="97c9a796-525c-464f-8f26-0317ae2657d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:11:01.024390 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:01.024345 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd"] Nov 01 23:11:01.026600 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:01.026573 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979beb82-2mrtd"] Nov 01 23:11:01.469420 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:01.469341 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd55e5c-c666-457a-8f5b-c53e7993683f" path="/var/lib/kubelet/pods/2bd55e5c-c666-457a-8f5b-c53e7993683f/volumes" Nov 01 23:11:15.587493 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:11:15.587461 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:11:28.417703 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:28.417671 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-548f455f69-d5qjg_02d479a1-5e3c-4d34-b486-4d8441c95f3a/authorino/0.log" Nov 01 23:11:32.671510 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:32.671476 6287 scope.go:122] "RemoveContainer" containerID="f3e28cd7d7171b832d55d5e48d8cbd0d1fdacd0d17c560de3183608023227c2c" Nov 01 23:11:32.679992 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:32.679970 6287 scope.go:122] "RemoveContainer" containerID="cec3dd78275ea63d1bf839fd379f8426ff345bdc25ddc2f3efa3bf32cc0329b0" Nov 01 23:11:38.340728 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:38.340693 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b6d674b9-llr7t_4459f450-7dab-4975-b2ac-02b042d1e004/manager/1.log" Nov 01 23:11:40.077417 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:40.077382 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b6d674b9-llr7t_4459f450-7dab-4975-b2ac-02b042d1e004/manager/0.log" Nov 01 23:11:40.782625 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:40.782591 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fdf88964b-zvq65_4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949/maas-api/0.log" Nov 01 23:11:41.598613 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:41.598585 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c44e4d6c-fflqb_e75d385f-258f-42e0-815b-375ba45f83f1/revoke-keys/0.log" Nov 01 23:11:43.307981 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:43.307947 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-548f455f69-d5qjg_02d479a1-5e3c-4d34-b486-4d8441c95f3a/authorino/0.log" Nov 01 23:11:43.708401 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:43.708327 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pv9xz_c74aadae-e2f9-47f6-b533-88919cce7c90/manager/0.log" Nov 01 23:11:44.454371 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:44.454340 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z26xx_bffc1a34-b42b-45e6-9350-31089c369391/manager/1.log" Nov 01 23:11:44.852083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:44.852048 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z26xx_bffc1a34-b42b-45e6-9350-31089c369391/manager/0.log" Nov 01 23:11:45.587875 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:11:45.587839 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:11:45.883407 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:45.883329 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d9489c85-swvcv_50a0b3cf-a357-4b31-8d99-65952baf4230/kube-auth-proxy/0.log" Nov 01 23:11:46.234600 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:46.234525 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-8m4jm_e65c3aec-3fe4-4c4f-8689-cbabab25e810/istio-proxy/0.log" Nov 01 23:11:46.750172 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:46.750105 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-fhcl9_1f464e54-af08-4804-a3a2-a4ff4ac3ef0d/istio-proxy/1.log" Nov 01 23:11:47.072978 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:47.072895 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-ncvpx_08d90fde-9034-4fa7-be99-57c1dfc881fd/istio-proxy/1.log" Nov 01 23:11:47.253898 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:47.253849 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-vt5hm_97c9a796-525c-464f-8f26-0317ae2657d7/istio-proxy/0.log" Nov 01 23:11:48.088408 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:48.088381 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx8f2_faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3/payload-processing/0.log" Nov 01 23:11:48.675573 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:48.675519 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-fhcl9_1f464e54-af08-4804-a3a2-a4ff4ac3ef0d/istio-proxy/0.log" Nov 01 23:11:49.013280 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:49.013254 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-ncvpx_08d90fde-9034-4fa7-be99-57c1dfc881fd/istio-proxy/0.log" Nov 01 23:11:49.843184 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:49.843157 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vvqpg_a7025b36-aa82-40c1-9fc1-e1e4470498c5/main/0.log" Nov 01 23:11:50.842492 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:50.842463 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg_e68fff20-1ed3-4f68-a012-444c11020805/main/0.log" Nov 01 23:11:51.668674 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:11:51.668639 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-548f455f69-d5qjg_02d479a1-5e3c-4d34-b486-4d8441c95f3a/authorino/0.log" Nov 01 23:12:00.219654 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:00.219622 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b6d674b9-llr7t_4459f450-7dab-4975-b2ac-02b042d1e004/manager/1.log" Nov 01 23:12:02.223524 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:02.223491 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fdf88964b-zvq65_4b6e458f-f5b8-4e9f-8cfb-2f7dceca8949/maas-api/0.log" Nov 01 23:12:03.080784 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:03.080751 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c44e4d6c-fflqb_e75d385f-258f-42e0-815b-375ba45f83f1/revoke-keys/0.log" Nov 01 23:12:04.503015 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:04.502949 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-548f455f69-d5qjg_02d479a1-5e3c-4d34-b486-4d8441c95f3a/authorino/0.log" Nov 01 23:12:04.901389 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:04.901306 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pv9xz_c74aadae-e2f9-47f6-b533-88919cce7c90/manager/0.log" Nov 01 23:12:05.582723 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:05.582696 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z26xx_bffc1a34-b42b-45e6-9350-31089c369391/manager/1.log" Nov 01 23:12:06.585360 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:06.585332 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d9489c85-swvcv_50a0b3cf-a357-4b31-8d99-65952baf4230/kube-auth-proxy/0.log" Nov 01 23:12:06.936450 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:06.936357 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-8m4jm_e65c3aec-3fe4-4c4f-8689-cbabab25e810/istio-proxy/0.log" Nov 01 23:12:07.451649 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:07.451609 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-fhcl9_1f464e54-af08-4804-a3a2-a4ff4ac3ef0d/istio-proxy/1.log" Nov 01 23:12:07.803416 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:07.803387 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-ncvpx_08d90fde-9034-4fa7-be99-57c1dfc881fd/istio-proxy/1.log" Nov 01 23:12:07.961415 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:07.961382 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-vt5hm_97c9a796-525c-464f-8f26-0317ae2657d7/istio-proxy/0.log" Nov 01 23:12:08.770699 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:08.770655 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx8f2_faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3/payload-processing/0.log" Nov 01 23:12:09.587980 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:09.587941 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vvqpg_a7025b36-aa82-40c1-9fc1-e1e4470498c5/main/0.log" Nov 01 23:12:10.559651 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:10.559620 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-x69sg_e68fff20-1ed3-4f68-a012-444c11020805/main/0.log" Nov 01 23:12:15.586814 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:12:15.586773 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:12:32.703712 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:32.703677 6287 scope.go:122] "RemoveContainer" containerID="d2d30b1b55145cb7931dbd904d03ce66212d166bf7c4389303f3edbbc44ecfb5" Nov 01 23:12:32.712653 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:32.712628 6287 scope.go:122] "RemoveContainer" containerID="145d3297dc631fe677e531f0807dbc8da6a983b98edc54690ffb5be01e8ab85c" Nov 01 23:12:42.607026 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:42.606989 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx8f2_faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3/payload-processing/0.log" Nov 01 23:12:42.771369 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:42.771339 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx8f2_faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3/payload-processing/0.log" Nov 01 23:12:43.022453 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:43.022419 6287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb"] Nov 01 23:12:43.025492 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:43.025463 6287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44e4d6c-fflqb"] Nov 01 23:12:43.469769 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:12:43.469689 6287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75d385f-258f-42e0-815b-375ba45f83f1" path="/var/lib/kubelet/pods/e75d385f-258f-42e0-815b-375ba45f83f1/volumes" Nov 01 23:12:45.588542 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:12:45.588494 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:13:15.587317 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:13:15.587276 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:13:17.465217 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:17.465178 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 23:13:17.465988 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:17.465969 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 23:13:19.868576 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:19.868552 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-node-driver-registrar/1.log" Nov 01 23:13:19.869309 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:19.869291 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-9kxlv_0e214ead-da27-47f8-8196-34d885b7b4dc/csi-driver/0.log" Nov 01 23:13:22.159489 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.159461 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx8f2_faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3/payload-processing/0.log" Nov 01 23:13:22.326125 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.326094 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx8f2_faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3/payload-processing/0.log" Nov 01 23:13:22.620848 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.620802 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w64nw/must-gather-th6fg"] Nov 01 23:13:22.621707 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.621680 6287 cpu_manager.go:414] "containerMap: removing container" podUID="555280ff-5432-484c-a622-640625058dce" containerName="maas-api" Nov 01 23:13:22.621845 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.621709 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555280ff-5432-484c-a622-640625058dce" containerName="maas-api" Nov 01 23:13:22.621845 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.621733 6287 cpu_manager.go:414] "containerMap: removing container" podUID="7f49282b-90a8-4ab9-954a-68996c610eea" containerName="payload-pre-processing" Nov 01 23:13:22.621845 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.621741 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f49282b-90a8-4ab9-954a-68996c610eea" containerName="payload-pre-processing" Nov 01 23:13:22.621845 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.621755 6287 cpu_manager.go:414] "containerMap: removing container" podUID="08c2d506-3b3c-40ab-9365-8dc11f484de7" containerName="payload-processing" Nov 01 23:13:22.621845 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.621762 6287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c2d506-3b3c-40ab-9365-8dc11f484de7" containerName="payload-processing" Nov 01 23:13:22.622140 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.621949 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="555280ff-5432-484c-a622-640625058dce" containerName="maas-api" Nov 01 23:13:22.622140 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.621961 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f49282b-90a8-4ab9-954a-68996c610eea" containerName="payload-pre-processing" Nov 01 23:13:22.622140 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.621979 6287 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c2d506-3b3c-40ab-9365-8dc11f484de7" containerName="payload-processing" Nov 01 23:13:22.626437 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.626403 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w64nw/must-gather-th6fg" Nov 01 23:13:22.632616 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.632580 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w64nw\"/\"openshift-service-ca.crt\"" Nov 01 23:13:22.632761 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.632631 6287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w64nw\"/\"kube-root-ca.crt\"" Nov 01 23:13:22.633842 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.633818 6287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w64nw\"/\"default-dockercfg-nlr9s\"" Nov 01 23:13:22.638427 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.638401 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w64nw/must-gather-th6fg"] Nov 01 23:13:22.768386 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.768348 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2d11d941-7766-4be9-9f89-3fb35de45507-must-gather-output\") pod \"must-gather-th6fg\" (UID: \"2d11d941-7766-4be9-9f89-3fb35de45507\") " pod="openshift-must-gather-w64nw/must-gather-th6fg" Nov 01 23:13:22.768601 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.768505 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86kh\" (UniqueName: \"kubernetes.io/projected/2d11d941-7766-4be9-9f89-3fb35de45507-kube-api-access-v86kh\") pod \"must-gather-th6fg\" (UID: \"2d11d941-7766-4be9-9f89-3fb35de45507\") " pod="openshift-must-gather-w64nw/must-gather-th6fg" Nov 01 23:13:22.869294 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.869257 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86kh\" (UniqueName: \"kubernetes.io/projected/2d11d941-7766-4be9-9f89-3fb35de45507-kube-api-access-v86kh\") pod \"must-gather-th6fg\" (UID: \"2d11d941-7766-4be9-9f89-3fb35de45507\") " pod="openshift-must-gather-w64nw/must-gather-th6fg" Nov 01 23:13:22.869502 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.869341 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2d11d941-7766-4be9-9f89-3fb35de45507-must-gather-output\") pod \"must-gather-th6fg\" (UID: \"2d11d941-7766-4be9-9f89-3fb35de45507\") " pod="openshift-must-gather-w64nw/must-gather-th6fg" Nov 01 23:13:22.869740 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.869716 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2d11d941-7766-4be9-9f89-3fb35de45507-must-gather-output\") pod \"must-gather-th6fg\" (UID: \"2d11d941-7766-4be9-9f89-3fb35de45507\") " pod="openshift-must-gather-w64nw/must-gather-th6fg" Nov 01 23:13:22.877284 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.877221 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86kh\" (UniqueName: \"kubernetes.io/projected/2d11d941-7766-4be9-9f89-3fb35de45507-kube-api-access-v86kh\") pod \"must-gather-th6fg\" (UID: \"2d11d941-7766-4be9-9f89-3fb35de45507\") " pod="openshift-must-gather-w64nw/must-gather-th6fg" Nov 01 23:13:22.941845 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:22.941812 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w64nw/must-gather-th6fg" Nov 01 23:13:23.215083 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:23.215004 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w64nw/must-gather-th6fg"] Nov 01 23:13:23.218024 ip-10-0-128-55 kubenswrapper[6287]: W1101 23:13:23.217991 6287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d11d941_7766_4be9_9f89_3fb35de45507.slice/crio-743ae6c33b9306805b50b2e9e89da16059b259db311970618e60da269cf9689f WatchSource:0}: Error finding container 743ae6c33b9306805b50b2e9e89da16059b259db311970618e60da269cf9689f: Status 404 returned error can't find the container with id 743ae6c33b9306805b50b2e9e89da16059b259db311970618e60da269cf9689f Nov 01 23:13:23.220392 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:23.220376 6287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:13:23.686270 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:23.686232 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w64nw/must-gather-th6fg" event={"ID":"2d11d941-7766-4be9-9f89-3fb35de45507","Type":"ContainerStarted","Data":"743ae6c33b9306805b50b2e9e89da16059b259db311970618e60da269cf9689f"} Nov 01 23:13:25.700591 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:25.700547 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w64nw/must-gather-th6fg" event={"ID":"2d11d941-7766-4be9-9f89-3fb35de45507","Type":"ContainerStarted","Data":"7ade4fb836bdbc93d45d425055628a16c2bedd6cf483b88445f53c114e24b0e2"} Nov 01 23:13:25.700591 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:25.700604 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w64nw/must-gather-th6fg" event={"ID":"2d11d941-7766-4be9-9f89-3fb35de45507","Type":"ContainerStarted","Data":"39d7e8f9ed63dcc51b777d78736da2d875d4c45e59c13228dac5d2d3602cdbe0"} Nov 01 23:13:25.719640 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:25.718534 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w64nw/must-gather-th6fg" podStartSLOduration=2.4638461400000002 podStartE2EDuration="3.718515643s" totalImagesPullingTime="1.254669503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:25.717396785 +0000 UTC m=+3018.365896026" watchObservedRunningTime="2026-11-01 23:13:25.718515643 +0000 UTC m=+3018.367014878" Nov 01 23:13:26.459940 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:26.459905 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cgcvk_330ca60e-5fa1-4a41-8617-73450d13cac2/global-pull-secret-syncer/0.log" Nov 01 23:13:26.725714 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:26.725636 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-prbmb_eba18759-4d3b-468a-9a93-0044457a0b93/konnectivity-agent/0.log" Nov 01 23:13:26.756217 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:26.756169 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-55.us-east-2.compute.internal_6ed9d7a120c3abfac5e4283e23adc9e4/haproxy/0.log" Nov 01 23:13:32.963180 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:32.962950 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-548f455f69-d5qjg_02d479a1-5e3c-4d34-b486-4d8441c95f3a/authorino/0.log" Nov 01 23:13:33.028160 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:33.028100 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pv9xz_c74aadae-e2f9-47f6-b533-88919cce7c90/manager/0.log" Nov 01 23:13:33.239322 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:33.239288 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z26xx_bffc1a34-b42b-45e6-9350-31089c369391/manager/0.log" Nov 01 23:13:33.255494 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:33.255470 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-z26xx_bffc1a34-b42b-45e6-9350-31089c369391/manager/1.log" Nov 01 23:13:35.039714 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.039626 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-xqjq8_7059b374-3864-4296-bd7b-d77161dfca28/monitoring-plugin/0.log" Nov 01 23:13:35.071869 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.071821 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8flmn_7d398858-b6ca-494f-a576-4456c4287992/node-exporter/0.log" Nov 01 23:13:35.096400 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.096371 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8flmn_7d398858-b6ca-494f-a576-4456c4287992/kube-rbac-proxy/0.log" Nov 01 23:13:35.120298 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.120271 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8flmn_7d398858-b6ca-494f-a576-4456c4287992/init-textfile/0.log" Nov 01 23:13:35.417798 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.417690 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/prometheus/0.log" Nov 01 23:13:35.441194 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.441148 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/config-reloader/0.log" Nov 01 23:13:35.466141 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.466084 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/thanos-sidecar/0.log" Nov 01 23:13:35.499687 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.499651 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/kube-rbac-proxy-web/0.log" Nov 01 23:13:35.539706 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.539669 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/kube-rbac-proxy/0.log" Nov 01 23:13:35.566034 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.566004 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/kube-rbac-proxy-thanos/0.log" Nov 01 23:13:35.594961 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.594924 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_97258e3b-22d2-487b-932c-d8dd9432d372/init-config-reloader/0.log" Nov 01 23:13:35.627674 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.627642 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-tpkmc_3b137d23-cef1-4c5b-975d-3902be4f9f83/prometheus-operator/0.log" Nov 01 23:13:35.648896 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:35.648868 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-tpkmc_3b137d23-cef1-4c5b-975d-3902be4f9f83/kube-rbac-proxy/0.log" Nov 01 23:13:36.337152 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.337093 6287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv"] Nov 01 23:13:36.352132 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.341771 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.355790 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.355755 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv"] Nov 01 23:13:36.441092 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.441052 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4cs\" (UniqueName: \"kubernetes.io/projected/bcfa0a29-ec08-4320-a0df-78ac15e2345f-kube-api-access-8z4cs\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.441092 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.441107 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-lib-modules\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.441393 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.441158 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-sys\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.441393 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.441193 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-proc\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.441393 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.441244 6287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-podres\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.541911 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.541874 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4cs\" (UniqueName: \"kubernetes.io/projected/bcfa0a29-ec08-4320-a0df-78ac15e2345f-kube-api-access-8z4cs\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.541911 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.541931 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-lib-modules\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.542224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.542035 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-sys\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.542224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.542082 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-sys\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.542224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.542096 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-proc\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.542224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.542041 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-lib-modules\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.542224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.542173 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-proc\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.542224 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.542185 6287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-podres\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.542566 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.542278 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bcfa0a29-ec08-4320-a0df-78ac15e2345f-podres\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.552672 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.552634 6287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4cs\" (UniqueName: \"kubernetes.io/projected/bcfa0a29-ec08-4320-a0df-78ac15e2345f-kube-api-access-8z4cs\") pod \"perf-node-gather-daemonset-hnllv\" (UID: \"bcfa0a29-ec08-4320-a0df-78ac15e2345f\") " pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.680440 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.680348 6287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:36.981540 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:36.980961 6287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv"] Nov 01 23:13:37.829004 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:37.828958 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" event={"ID":"bcfa0a29-ec08-4320-a0df-78ac15e2345f","Type":"ContainerStarted","Data":"eccd28fd36c82620327b406b33b9eac13e365ae4fbd70335ce23df351363fb7e"} Nov 01 23:13:37.829419 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:37.829066 6287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" event={"ID":"bcfa0a29-ec08-4320-a0df-78ac15e2345f","Type":"ContainerStarted","Data":"f25c700662df980ca8fbece0a9505617d1f44469ce57ee0ace4b8cda758ebd8d"} Nov 01 23:13:37.829419 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:37.829235 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:37.846423 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:37.846372 6287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" podStartSLOduration=1.846356138 podStartE2EDuration="1.846356138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:37.845473134 +0000 UTC m=+3030.493972367" watchObservedRunningTime="2026-11-01 23:13:37.846356138 +0000 UTC m=+3030.494855364" Nov 01 23:13:38.343046 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:38.343013 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5fdc5c4886-h5qt9_49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26/console/1.log" Nov 01 23:13:38.343644 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:38.343607 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5fdc5c4886-h5qt9_49e9bbfa-4014-4bd1-8ba1-f00d8fa0da26/console/0.log" Nov 01 23:13:40.079874 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:40.079846 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-78mfv_8c95cb60-d629-48ef-859d-85d7893afe81/dns/0.log" Nov 01 23:13:40.103662 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:40.103638 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-78mfv_8c95cb60-d629-48ef-859d-85d7893afe81/kube-rbac-proxy/0.log" Nov 01 23:13:40.327708 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:40.327668 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9pqxl_5265e1c6-2a08-4439-b6a9-42ed2fdbd81a/dns-node-resolver/0.log" Nov 01 23:13:41.168567 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:41.168538 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wbfqp_cdc57449-d2a8-41fa-808b-6ef9a3517bc6/node-ca/0.log" Nov 01 23:13:43.848140 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:43.848095 6287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w64nw/perf-node-gather-daemonset-hnllv" Nov 01 23:13:45.336708 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:45.336676 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d9489c85-swvcv_50a0b3cf-a357-4b31-8d99-65952baf4230/kube-auth-proxy/0.log" Nov 01 23:13:45.416526 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:45.416498 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-8m4jm_e65c3aec-3fe4-4c4f-8689-cbabab25e810/istio-proxy/0.log" Nov 01 23:13:45.523884 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:45.523849 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-fhcl9_1f464e54-af08-4804-a3a2-a4ff4ac3ef0d/istio-proxy/0.log" Nov 01 23:13:45.524051 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:45.523941 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-fhcl9_1f464e54-af08-4804-a3a2-a4ff4ac3ef0d/istio-proxy/1.log" Nov 01 23:13:45.582326 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:45.582297 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-ncvpx_08d90fde-9034-4fa7-be99-57c1dfc881fd/istio-proxy/0.log" Nov 01 23:13:45.584523 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:45.584453 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-ncvpx_08d90fde-9034-4fa7-be99-57c1dfc881fd/istio-proxy/1.log" Nov 01 23:13:45.611440 ip-10-0-128-55 kubenswrapper[6287]: E1101 23:13:45.611285 6287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b6vcp" containerName="extractor" Nov 01 23:13:45.627007 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:45.626977 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-vt5hm_97c9a796-525c-464f-8f26-0317ae2657d7/istio-proxy/0.log" Nov 01 23:13:45.790701 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:45.790670 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx8f2_faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3/payload-processing/0.log" Nov 01 23:13:48.049673 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:48.049633 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9xvw2_f1a395fe-2467-4736-9fd3-bab535023c24/serve-healthcheck-canary/0.log" Nov 01 23:13:48.552829 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:48.552799 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b6vcp_800f0c00-7f68-419a-96c5-6b947a2101a5/kube-rbac-proxy/0.log" Nov 01 23:13:48.578126 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:48.578085 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b6vcp_800f0c00-7f68-419a-96c5-6b947a2101a5/exporter/0.log" Nov 01 23:13:48.605339 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:48.605308 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b6vcp_800f0c00-7f68-419a-96c5-6b947a2101a5/extractor/0.log" Nov 01 23:13:49.669658 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:49.669628 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d9489c85-swvcv_50a0b3cf-a357-4b31-8d99-65952baf4230/kube-auth-proxy/0.log" Nov 01 23:13:49.712494 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:49.712464 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-8m4jm_e65c3aec-3fe4-4c4f-8689-cbabab25e810/istio-proxy/0.log" Nov 01 23:13:49.777907 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:49.777867 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-fhcl9_1f464e54-af08-4804-a3a2-a4ff4ac3ef0d/istio-proxy/0.log" Nov 01 23:13:49.782284 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:49.782259 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-fhcl9_1f464e54-af08-4804-a3a2-a4ff4ac3ef0d/istio-proxy/1.log" Nov 01 23:13:49.814145 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:49.814108 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-ncvpx_08d90fde-9034-4fa7-be99-57c1dfc881fd/istio-proxy/1.log" Nov 01 23:13:49.814287 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:49.814108 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-ncvpx_08d90fde-9034-4fa7-be99-57c1dfc881fd/istio-proxy/0.log" Nov 01 23:13:49.826834 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:49.826806 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-vt5hm_97c9a796-525c-464f-8f26-0317ae2657d7/istio-proxy/0.log" Nov 01 23:13:49.909914 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:49.909883 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rx8f2_faf889a0-6a6a-4ecd-98b1-fd2b2d550ce3/payload-processing/0.log" Nov 01 23:13:51.417480 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:51.417442 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8487c6b6d6-r86kq_c50a9903-80f6-457c-bcf9-9838a3cbdd64/manager/0.log" Nov 01 23:13:51.440278 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:51.440249 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8487c6b6d6-r86kq_c50a9903-80f6-457c-bcf9-9838a3cbdd64/copy-manifests/0.log" Nov 01 23:13:52.665007 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:52.664980 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-76ffc984cf-cpvzx_3633d07f-afe4-4477-bc86-d1881c6db757/manager/1.log" Nov 01 23:13:52.669007 ip-10-0-128-55 kubenswrapper[6287]: I1101 23:13:52.668949 6287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-76ffc984cf-cpvzx_3633d07f-afe4-4477-bc86-d1881c6db757/manager/0.log"